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METHODOLOGY  FOR  EVALUATING  TRANSFER  OF  LEARNING  FROM  THE  U.S. 
ARMY’S  ADVANCED  LEADERS  COURSE 

EXECUTIVE  SUMMARY 


Research  Requirement: 

As  part  of  its  institutional  transformation,  the  U.S.  Army  Infantry  School  has  revised  the 
program  of  instruction  (POI)  for  the  resident  Infantry  Advanced  Leaders  Course  (ALC).  As  the 
new  POI  enters  implementation,  empirical  feedback  is  needed  to  ensure  program  goals  are  being 
met,  with  a  focus  on  operational  unit  impact.  The  feedback  can  be  used  to  refine  the  course  POI 
and  enhance  the  program’s  contributions  to  unit  readiness.  Under  the  sponsorship  of  the  Infantry 
School,  the  U.S.  Army  Research  Institute  (ARI)  initiated  a  program  to  evaluate  the  transfer  of 
Infantry  ALC  training  to  job  performance  and  unit  operations.  The  current  research  aimed  to 
establish  the  planning  foundation  for  the  evaluation. 

Procedure: 

The  researchers  used  an  iterative  develop-review-revise  process  to  create  a  family  of 
planning  documents  within  a  program  evaluation  framework.  They  began  by  developing  metrics 
of  Infantry  ALC  impact,  then  crafted  an  evaluation  plan,  and  finally  constructed  tools  and 
guidelines  for  collecting  data.  Interviews  of  ALC  program  players  yielded  well-grounded  input 
to  the  metrics.  The  team  used  the  metrics  to  shape  the  evaluation  objectives  and  to  design  the 
data  collection  instruments.  Robust  internal  reviews  drove  iterative  revision  efforts. 

Findings: 

The  planning  documents  include  an  Evaluation  Design  Plan,  a  Data  Collection  and 
Management  Plan,  measures  of  ALC  impact,  the  architecture  for  a  coherent  set  of  data  collection 
instruments,  and  a  Data  Collector’s  Guide.  As  a  result  of  the  iterative  develop-review-revise 
process,  the  products  provide  defensible  evaluation  enablers  that  reflect  realistic  planning 
parameters  and  constraints.  The  presentation  summarizes  design  options  considered  and 
rejected,  and  delineates  the  assumptions  behind  the  data  collection  strategy.  The  intermediate 
products  generated  during  development  of  the  impact  measures  offer  diverse  perspectives  on  the 
challenges  of  measuring  transfer  of  training  in  an  operational  unit  context.  The  primary  products 
are  included  in  appendixes. 

Utilization  and  Dissemination  of  Findings: 

The  products  of  this  research  provide  the  enablers  for  a  comprehensive  evaluation  of 
Infantry  ALC’s  operational  impact.  As  the  follow-on  team  prepares  for  data  collection,  they 
must  update  and  finalize  the  products,  complete  the  development  of  data  collection  instruments, 
and  conduct  a  thorough  pilot  test.  The  eventual  results  of  the  evaluation  will  help  decision 
makers,  course  managers,  and  curriculum  developers  to  refocus  training  so  as  to  enhance  the 
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transfer  of  ALC-acquired  knowledge  and  skills.  In  addition,  the  impact  metrics  may  prove 
useful  in  career  fields  beyond  Infantry. 
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Methodology  for  Evaluating  Transfer  of  Learning  from  the 
U.S.  Army’s  Advance  Leaders  Course 


Introduction 

The  U.S.  Army  is  currently  in  the  process  of  transforming  its  Noncommissioned  Officer 
Education  System  (NCOES)  to  meet  the  needs  of  the  operational  Army  and  ensure  relevance  to 
present  and  future  operations.  A  number  of  factors  driven  by  the  contemporary  operational 
environment  (COE)  have  shaped  the  direction  of  the  transformation,  such  as  a  more  complex 
array  of  threats,  new  requirements  for  proficiency  on  unfamiliar  tasks,  and  persistently  high 
operational  tempo  (OPTEMPO).  As  part  of  the  NCOES  transformation,  the  U.S.  Army  Infantry 
School  has  revised  its  program  of  instruction  (POI)  for  the  resident  Infantry  Advanced  Leaders 
Course  (ALC),  with  emphasis  on  the  resident  Phase  II  POI.  As  the  new  POI  is  finalized  and 
implemented,  empirical  feedback  is  needed  to  ensure  program  goals  are  being  met.  The 
feedback  should  focus  on  noncommissioned  officer  (NCO)  competence/performance  and  unit 
readiness  in  the  context  of  the  COE.  The  feedback  can  then  be  used  to  refine  the  ALC  POI  and 
enhance  the  program’s  contributions  to  unit  readiness.  Under  the  sponsorship  of  the  Infantry 
School,  the  U.S.  Army  Research  Institute  (ARI)  initiated  a  program  to  examine  the  operational 
impact  of  Infantry  ALC  training.  The  current  research  was  conducted  by  ARI’s  Fort  Benning 
Research  Unit  to  create  the  methodological  foundation  for  evaluating  the  impact  of  the  resident 
Infantry  ALC  Phase  II  POI  on  Soldier  perfonnance  and  readiness  in  operational  units. 

This  publication  presents  the  planning  and  preparation  products  developed  to  support  a 
comprehensive  evaluation  of  Infantry  ALC’s  impact  on  operational  parameters.  The  main  body 
details  the  methods  used,  then  describes  and  characterizes  the  products.  The  primary  products 
themselves  are  provided  in  appendixes.  The  infonnation  and  products  are  of  use  to  the  research 
team  that  will  conduct  the  evaluation  as  well  as  investigators  working  to  evaluate  other 
instructional  programs. 


Background 

In  support  of  its  modular  force  initiatives,  the  U.S.  Army  is  transforming  the  NCOES  to 
meet  the  compelling  realities  of  the  COE.  An  important  goal  is  to  ensure  that  the  various 
NCOES  programs  are  relevant  to  present  and  future  operations.  A  number  of  factors  from  the 
COE  have  shaped  the  direction  of  the  emerging  NCOES — especially  the  transition  to  a  modular 
force  driven  by  the  Army  Force  Generation  (ARFORGEN)  process,  a  more  complex  and 
uncertain  COE,  the  expanding  number  of  experienced/senior  NCOs,  new  training  requirements, 
and  the  persistently  heightened  OPTEMPO.  These  factors  and  others  are  reshaping  the  structure, 
approaches,  and  content  of  key  POIs. 

A  critical  objective  of  the  NCOES  transformation  is  to  ensure  the  programs  develop 
competent  and  adaptive  leaders  required  in  a  more  complex  and  uncertain  environment.  The 
COE  is  changing  constantly,  and  the  NCOES  faces  challenges  to  keep  pace.  Army  NCOs  will 
continue  to  be  the  experts  in  leader  tasks  for  their  respective  levels  of  responsibility  and  in 
individual  and  small-unit  training.  They  will  continue  to  be  the  recognized  experts  in  field  craft, 
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marksmanship.  Soldier  care  and  technical  skills.  In  addition  to  these  traditional  skills,  the  Army 
must  develop  NCOs  who  can  digest  new  information  quickly,  adapt  to  rapid  mission  changes, 
and  take  advantage  of  opportunities  on  the  battlefield.  The  NCOES  aims  to  train  the  right  tasks 
at  the  right  levels  and  to  prepare  NCOs  to  operate  in  both  analog  and  digital  environments. 

Within  the  newly  emerging  NCOES  family  of  programs,  the  ALC  program  aims  to 
develop  squad/section  leaders  who  are  masters  of  their  military  occupational  specialties  (MOSs) 
as  well  as  expert  trainers  and  training  managers.  The  ALC  POI  (formerly  known  as  the  Basic 
Noncommissioned  Officer  Course,  or  BNCOC)  focuses  on  leading  and  training  within  the 
platoon  formation  and  becoming  familiar  with  core  staff  skills  needed  within  the  battalion.  The 
ALC  program  has  undergone  substantive  changes,  including  increased  emphasis  on  student 
involvement  in  training,  clearer  focus  on  leading  and  training  within  platoons/squads,  multi¬ 
echelon  shared  training  events,  and  exposure  to  staff  skills.  The  coursework  trains  warfighting 
skills,  combat  leadership,  and  training  fundamentals.  The  common  core  portion  of  the  course 
(Phase  I)  is  taught  via  distance  learning  (video  tele-training)  or  in  residence  (by  exception)  at  an 
NCO  Academy  (NCOA).  The  technical  portion  of  the  course  (Phase  II)  is  completed  in 
residence  and  lasts  4-5  weeks,  depending  on  the  career  field.  A  key  program  goal  is  to  conduct 
all  training  in  a  small  group  instruction  environment  with  an  instructor-to-student  ratio  of  one 
instructor  per  12-16  students. 

The  current  OPTEMPO  has  also  impacted  Infantry  ALC  from  other  perspectives  as  well. 
For  example,  suspending  the  requirement  for  Soldiers  to  complete  ALC  training  prior  to 
promotion  to  Staff  Sergeant  (SSG)  allows  Soldiers  to  advance  with  the  stipulation  that  training 
be  completed  later,  usually  within  180  days  after  completing  deployment.  Thus,  many  Soldiers 
enter  ALC  having  already  served  as  squad  leaders  (in  combat).  This  has  resulted  in  a  greater 
diversity  in  rank  and  experience  for  Soldiers  attending  Infantry  ALC  training — anywhere  from 
E5  (SGT)  to  E7  (SFC).  While  most  ALC  attendees  have  combat  experience,  a  number  have  not 
had  unit  troop  leading  experience  since  being  fire  team  leaders,  while  others  have  been  Platoon 
Sergeants  in  combat.  This  also  presents  a  serious  challenge  to  course  developers.  With  Infantry 
ALC  training  time  cut  from  slightly  over  six  weeks  to  four  weeks,  course  developers  must  make 
difficult  curriculum  decisions  concerning  what  topic  areas  to  include,  level  of  detail,  method  of 
presentation,  and  how  to  enhance  course  relevance.  In  addition,  the  Infantry  School  must 
prepare  instructors  to  teach  more  advanced  topics  that  are  being  migrated  from  the  Advanced 
Noncommissioned  Officer  Course  (ANCOC),  such  as  composite  risk  assessment,  military 
decision  making  process,  mission  planning,  and  orders  preparation  and  presentation. 

There  is  a  need  to  assess  the  effectiveness  of  the  new  Infantry  ALC  Phase  II  POI  in  terms 
of  its  impact  on  Soldier  performance/readiness  in  operational  units.  While  gains  observed  in 
interim  tests  and  end-of-course  performance  scores  are  suggestive,  these  improvements  must  be 
meaningful  and  linked  to  enhanced  Soldier  performance/readiness  in  the  COE.  Information  on 
operational  impact  is  needed  to  enable  decision  makers,  course  managers,  and  POI  developers  to 
refocus  training  where  areas  are  identified  for  improvement.  In  particular,  course  developers 
must  understand  the  impact  of  the  curriculum  decisions  noted  earlier  on  the  Soldier’s  ability  to 
apply — in  other  words,  transfer — the  knowledge  and  skills  acquired  in  Infantry  ALC  to  real 
world  environments. 
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Transfer  of  Training 


Framework  and  Definitions 

Transfer  of  training  generally  refers  to  the  application  of  acquired  knowledge  and  skills 
to  job  performance  (Burke  &  Hutchins,  2007).  According  to  Baldwin  and  Ford  (1988),  two 
conditions  must  be  met  for  transfer  to  occur — learned  behavior  must  (a)  generalize  to  the  job 
context  and  (b)  sustain  itself  over  a  period  of  time  on  the  job.  Smith,  Ford,  and  Kozlowski 
(1997)  argue  for  a  third  condition  or  indicator  of  transfer,  the  extent  to  which  the  trainee  can 
adapt  to  novel  or  changing  situational  demands,  with  an  emphasis  on  developing  effective 
problem  solving  skills.  The  present  research  adopted  these  three  conditions  as  the  framework  for 
defining  transfer  of  training.  Given  the  ultimate  purpose  of  the  Infantry  ALC  POI,  the  research 
focused  on  the  impact  of  the  resident  phase  on  operational  readiness.  This  focus  led  to  selection 
of  key  dimensions — leadership,  job  perfonnance  and  unit  readiness — that  could  point  to  markers 
of  ALC’s  impact  on  operationally  relevant  parameters. 

The  interest  in  transferring  skills  from  military  training  to  operational  settings  is  more 
than  academic  when  survival  in  combat  is  at  stake.  Transfer  should  be  a  primary  goal  of  Army 
training  programs.  Recent  studies,  however,  indicated  that  the  impact  of  training  programs  on 
transfer  has  been  quite  limited.  Based  on  survey  methods,  Saks  (2002)  reported  that  about  40% 
of  trainees  fail  to  transfer  immediately  after  training,  70%  falter  in  transfer  one  year  after  the 
program,  and  ultimately  only  50%  of  training  investments  result  in  organizational  or  individual 
improvements.  In  large  part,  the  poor  return  on  investment  can  be  attributed  to  the  failure  to  (a) 
systematically  assess  intervening/contextual  variables  impacting  the  transfer  process  (Holton, 
Bates,  &  Ruona,  2000)  and  (b)  adjust  programs  to  improve  the  training  transfer  process. 

Near  transfer  is  the  application  of  learning  to  on-the-job  situations  similar  to  those  in 
which  initial  learning  occurred  (e.g.,  repairing  a  radio).  Near  transfer  would  be  the  objective  of 
short-term  skill  development  that  can  be  applied  immediately  to  improve  performance  of  one’s 
current  duties  (Spitzer,  1984).  It  is  most  critical  in  technical  training,  where  typically  the  goal  is 
to  teach  specific  behaviors  and  procedures — such  as  operating  a  machine  lathe  or  repairing  small 
engines — that  are  applicable  to  the  individual’s  present  job  (Laker,  1990). 

In  contrast,  far  transfer  refers  to  applying  learning  to  situations  dissimilar  to  those  of  the 
original  learning  (e.g.,  problem  solving).  Far  transfer  involves  applying  general  principles  so  the 
learner  can  solve  problems  in  the  transfer  environment  (Goldstein,  1986).  This  approach  is  most 
appropriate  for  higher  level  cognitive  skills  such  as  decision-making  and  creative  problem 
solving  (adaptability),  where  the  focus  is  on  preparing  trainees  to  better  deal  with  unspecified 
problem  areas  in  the  future.  (For  further  discussion  of  near  versus  far  transfer,  see  Barnett  & 
Ceci,  2002,  and  Kim  &  Lee,  2001.) 

Training  for  Transfer 

As  part  of  the  objective/goal  setting  process,  trainers  must  determine  if  the  training  aims 
to  improve  near  transfer,  far  transfer,  or  both  (Yamnill  &  McLean,  2001).  Near  and  far  transfer 
can  be  viewed  as  a  series  of  goals  or  objectives  of  training  and  should  be  reflected  in  the  design 
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and  content  of  instruction.  It  is  critical  to  identify  in  advance  the  situations  in  which  training  is 
to  be  applied.  No  matter  how  well  designed  the  training,  the  instructional  program  must  directly 
address  clearly  identified  individual  and  organizational  problems.  Thus,  the  extent  to  which  the 
knowledge,  skills,  and  attributes  taught  in  training  are  relevant  to  the  trainee’s  job  perfonnance  is 
critical  in  detennining  transfer  (Yamnill  &  McLean,  2001). 

Transfer  climate.  Transfer  of  learning  is  a  complex  process  and  involves  multiple  factors 
and  influences.  The  transfer  climate  may  either  support  or  inhibit  application  of  learning  on  the 
job  and  is  viewed  by  some  (e.g.,  Rouiller  &  Goldstein,  1993)  to  be  at  least  as  important  in 
facilitating  transfer  as  learning  itself.  From  a  training  investment  perspective,  it  is  important  that 
the  trainer  or  practitioner  have  the  tools  to  accurately  diagnose  those  factors  inhibiting  the 
transfer  process  and  then  intervene  where  appropriate  (Holton,  Bates,  &  Ruona,  2000). 

Kirkpatrick’s  four-level  evaluation  model.  Evaluation  of  interventions  is  one  of  the  most 
critical  issues  faced  by  training  practitioners  in  the  area  of  human  resource  development.  The 
dominant  evaluation  model  currently  in  use  is  the  one  developed  by  Kirkpatrick  (1994).  While 
the  model  has  certain  shortcomings  (see  Holton,  1996)  it  nevertheless  provides  a  useful  means  of 
categorizing  data.  Kirkpatrick’s  (1994)  evaluation  model  defines  four  levels — reaction,  learning, 
behavior,  and  results.  These  levels  are  hierarchical,  progressing  from  trainee  reactions  to 
organizational  outcomes.  Briefly,  reaction  refers  to  how  the  trainees  react  to  the  training, 
characterized  by  overall  like/ dislike,  acceptability,  perceived  quality,  and  similar  dimensions. 
Learning  is  the  extent  to  which  trainees  change  attitudes,  improve  knowledge,  and/or  increase 
skills  as  a  result  of  attending  a  program.  Behavior  is  defined  as  the  extent  to  which  change  in  job 
performance  (transfer  of  learning  to  the  work  place)  has  occurred  because  the  participant 
received  the  training.  Results  refer  to  the  final  objectives  of  the  program,  such  as  better  unit 
readiness,  decreased  equipment  failures,  reduced  frequency/severity  of  accidents,  and  improved 
unit  readiness.  From  one  level  to  the  next,  the  evaluation  process  becomes  more  difficult  and 
time  consuming,  but  the  payoff  in  information  obtained  can  be  extremely  valuable. 

Current  Research  Strategy 

The  present  research  incorporated  Kirkpatrick’s  taxonomy  to  guide  the  evaluation 
strategy.  Measurement  was  targeted  at  assessing  reactions  (e.g.,  surveys  of  student  expectations/ 
attitudes,  course  characteristics,  and  job  relevance),  learning  (subjective  and  objective  measures 
of  NCO  competencies  across  ALC  tasks),  behavior  (e.g.,  individual  and  unit  performance  from 
surveys,  unit  records,  and  focus  group  interviews),  and  results  (e.g.,  unit  readiness,  morale  and 
benefits  from  surveys  and  focus  group  interviews).  The  measurement  focus  embraced  both  near 
transfer  (job-relevant  abilities,  behaviors,  and  attitudes)  and  indicators  of  far  transfer  (individual 
and  unit  perfonnance,  mission  accomplishment,  unit  readiness,  etc.). 

Measures  of  transfer  climate  (global  measures  such  as  unit  conditions  influencing  NCOs’ 
application  of  ALC  learning)  were  also  targeted  for  the  present  research.  The  measurement 
strategy  extended  to  key  dimensions  relating  to  personnel,  training,  and  organizational  factors 
that  influence  transfer  of  learning  to  job  perfonnance  (Holton,  Bates,  and  Ruona,  2000).  These 
contributing  factors  were  intended  to  enhance  the  value  of  Kirkpatrick’s  evaluation  model  by 
providing  information  on  dimensions  affecting  transfer,  thus  helping  trainers/researchers  move 


4 


beyond  the  question  of  whether  the  training  works  to  a  deeper  understanding  of  why  the  training 
was  effective.  In  summary,  the  research  strategy  incorporated  central  issues  identified  by  other 
researchers  (e.g.,  Burke  &  Hutchins,  2007;  Holton,  Bates  &  Ruona,  2000)  to  improve  the 
evaluation  process.  More  specifically,  the  strategy  targeted  the  following:  (a)  multisource 
feedback  from  trainees/peers,  leaders,  and  unit  records,  (b)  assessment  of  training  and  work 
environment  factors  impacting  the  transfer  process,  and  (c)  extension  of  the  transfer  interval  to 
allow  for  assimilation  and  practice  of  newly  acquired  skills. 

The  importance  of  detennining  the  impact  of  the  resident  Infantry  ALC  on  operational 
readiness  is  obvious.  A  comprehensive  evaluation  focusing  on  both  near  and  far  transfer  effects 
is  critical  to  identifying  and  understanding  issues  shaping  the  NCOES  transformation  as  well  as 
determining  the  extent  to  which  NCOES  objectives  are  being  met. 

Technical  Objectives 

The  overarching  goal  of  the  current  project  was  to  establish  the  methodological  enablers 
for  evaluating  the  training  effectiveness  of  the  revised  Infantry  ALC  Phase  II  POI.  These 
enablers  included  evaluation  planning  documents,  metrics  of  operational  impact,  data  collection 
tools,  and  evaluation  procedures.  By  leveraging  a  program  evaluation  approach  as  the  technical 
foundation  (e.g.,  Kirkpatrick,  1994;  Holton,  1996),  the  research  set  the  stage  for  assessing  the 
impact  of  the  resident  Infantry  ALC  Phase  II  POI  on  operational  parameters  of  tactical  units. 

The  ultimate  aim  was  to  link  ALC  learning  to  individual  and  group  perfonnance  as  well  as  unit 
readiness.  The  following  technical  objectives  guided  the  research: 

•  Establish  key  behaviors  and  candidate  metrics  of  Infantry  ALC  Phase  II  impact  on 
Soldier  performance  and  readiness  in  operational  units. 

•  Develop  a  field  evaluation  design  with  sampling  plan  and  supporting  materials. 

•  Construct  suitable  data  collection  instruments  for  use  in  schoolhouse  and  units. 

•  Develop  procedures  for  using  the  instruments  to  collect  training  impact  data. 

The  results  of  this  research  represent  the  first  step  in  determining  the  new  Infantry  ALC 
Phase  II  POI’s  impact.  By  creating  the  design,  methodology  and  tools  for  a  unit- focused 
evaluation,  the  research  paved  the  way  for  follow-on  efforts  to  collect  comprehensive  data  in  the 
field. 


Method 

This  section  explains  the  method  used  to  develop  the  various  products  (enablers)  for  the 
planned  evaluation.  Tailored  processes  were  employed,  depending  on  the  outcome  required. 
The  technical  approach  called  for  developing  planning  and  execution  products  within  a  program 
evaluation  framework  (Kirkpatrick,  1994;  Holton,  1996).  The  research  team  began  by  crafting 
metrics  of  ALC  impact,  then  developed  an  evaluation  plan,  and  finally  constructed  tools  and 
guidelines  for  collecting  data.  Interviews  of  Infantry  ALC  program  players  yielded  constructive 
input  to  the  metrics.  The  team  used  the  metrics  to  shape  the  evaluation  objectives  and  also  to 
design  the  data  collection  instruments.  A  robust,  iterative  internal  review  process  helped  the 
team  reach  a  state  of  consensus  on  each  product. 
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The  following  major  phases  were  executed  to  design,  develop,  and  refine  the  various 
enablers  of  the  follow-on  evaluation: 

•  Creation  of  measures  based  on  prioritized  behavioral  and  performance  outcomes 

•  Thorough  evaluation  design,  planning  and  documentation 

•  Design  of  a  coherent  family  of  data  collection  instruments 

•  Development,  review,  and  refinement  of  selected  data  collection  instruments 

•  Development  of  data  collection  procedures  in  the  form  of  a  user  friendly  guide 

Development  of  Course  Impact  Metrics 

The  team  began  by  reviewing  the  course  outline  for  Infantry  ALC  Phase  II.  The  modules 
and  lessons  listed  in  the  course  outline  were  analyzed  to  produce  a  detailed  list  of  behavioral 
outcomes  (end  state  behaviors  such  as  incorporating  land  navigation  into  training  plans).  The 
analysis  also  considered  a  small-unit  proficiency  test  plan  used  by  ALC  instructors.  The 
behavioral  outcomes  were  not  limited  to  course  learning  objectives,  but  rather  encompassed  a 
broader  spectrum  of  student  behaviors  implied  by  the  instructional  topics.  From  the  list  of 
behavioral  outcomes  the  team  created  a  behavioral  taxonomy  consisting  of  functional  categories 
(e.g.,  Warrior  proficiency,  direct  leadership)  with  sample  behaviors  (e.g.,  perfonning  combative 
techniques  to  standard)  for  each  category. 

In  parallel  with  the  analysis  of  Infantry  ALC  behavioral  outcomes,  the  team’s  subject 
matter  experts  (SMEs)  reviewed  Anny  publications  to  glean  doctrinal  competencies  expected  of 
squad  leaders  (U.S.  Department  of  the  Army,  2002,  2006a,  2006b,  2007,  2008).  The  review 
produced  a  list  of  leadership,  job  performance  and  unit  readiness  indicators — such  as  fostering 
teamwork  and  cohesion — that  could  serve  as  markers  of  ALC’s  impact  on  operationally  relevant 
parameters.  Collectively  the  competence  indicators  fonned  a  picture  of  a  squad  leader’s 
expected  performance  parameters  from  a  tactical  unit’s  perspective. 

The  team  next  merged  the  behavioral  taxonomy  with  the  indicators  of  squad  leader 
competence  to  produce  a  candidate  list  of  ALC  impact  metrics.  In  the  initial  list,  a  modest 
family  of  measures  (e.g.,  NCO  conducts  timely  after  action  reviews,  or  AARs)  was  organized 
under  a  small  set  of  functional  categories  (e.g.,  unit  training,  resource  management).  During 
structured  interviews  with  Infantry  ALC  representatives  (NCOA  Commandant,  course  manager, 
and  senior  instructors),  the  team  presented  and  discussed  the  initial  list  of  metrics.  The  resulting 
feedback  led  the  team  to  revise  the  list  around  the  five  roles  of  a  multi-functional  NCO — critical 
and  creative  thinker,  warrior  leader,  leader  developer,  ambassador,  and  resource  manager 
(Infantry  ALC  Course  Manager,  personal  communication,  May  7,  2008).  After  a  series  of 
revise-review-refine  cycles  the  final  version  of  the  metrics  list  emerged  with  a  streamlined 
family  of  measures,  level-2  categories,  and  an  extra  category  to  account  for  global  impact.  A 
final  interview  with  two  course  graduates  yielded  suggestions  for  refining  and  prioritizing  the 
measures. 

In  developing  the  measures  of  global  impact,  the  team  capitalized  on  published  work  by 
Facteau,  Dobbins,  Russell,  Ladd,  and  Kudisch  (1995),  Crow  (2007),  and  Holton,  Bates,  and 
Ruona  (2000).  These  investigators  identified  individual  and  organizational  dimensions  that  are 
important  for  understanding  or  explaining  training  transfer  effects  (e.g.,  motivation,  incentives, 
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organizational  commitment,  supervisor’s  support).  These  dimensions  helped  shape  the  global 
impact  measures,  with  an  emphasis  on  perceptions  revealed  by  self-ratings. 

The  final  step  in  creating  the  Infantry  ALC  impact  metrics  entailed  defining  data 
elements  for  each  measure,  yielding  an  additional  level  of  detail.  As  an  example,  the  measure 
for  fostering  teamwork/positive  climate  spawned  six  data  elements  (e.g.,  adverse  personnel 
incidents  such  as  unauthorized  absence).  The  aggregated  measures  with  their  data  elements  were 
compiled  into  a  master  list  that  could  later  guide  the  development  of  data  collection  instruments. 

Development  of  Evaluation  Plan 

A  primary  goal  of  the  project  was  to  create  a  scientifically  defensible  framework  and 
methodology  for  collecting  evaluation  data.  The  team  developed  two  principal  planning 
documents:  (a)  the  Evaluation  Design  Plan  and  (b)  the  Data  Collection  and  Management  Plan. 
The  investigators  employed  sequential,  top-down  design  and  development  processes  to  ensure 
accurate  functional  linkages  between  the  two  documents. 

The  process  for  developing  the  Evaluation  Design  Plan  balanced  program  evaluation  best 
practices  (Kirkpatrick,  1994;  Holton,  1996)  with  the  constraints  of  field  research.  The  team 
began  by  defining  the  evaluation’s  purpose,  goals,  and  objectives.  They  then  analyzed  those 
driving  factors  to  determine  key  design  parameters  such  as  investigational  paradigm,  target 
population,  design  limitations,  measurement  focus,  and  data  collection  environment.  Next  the 
team  developed  an  evaluation  approach  deemed  suitable  for  satisfying  the  objectives  and  design 
parameters.  Given  the  defined  target  population  and  the  evaluation  limitations,  they  crafted  a 
sampling  strategy  aimed  at  yielding  defensible  sample  sizes  for  the  various  sources  of  data. 

After  documenting  the  critical  assumptions  underlying  the  planned  evaluation,  the  team 
prepared  a  comprehensive  measurement  strategy  encompassing  efficient  processes  for  capturing 
robust,  high-quality  data.  The  measurement  strategy  extrapolated  from  the  ALC  impact  metrics 
to  define  the  types  of  measures  (primary  and  secondary).  The  next  step  entailed  identifying  the 
family  of  required  data  collection  instruments  and  data  sources.  The  final  step  created  a  start-to- 
finish  timeline,  by  stage,  for  the  data  collection  activities  and  built  a  straw-man  schedule 
accounting  for  the  various  stages. 

Incorporated  in  the  development  method  was  a  series  of  internal  review  steps,  each  of 
which  prompted  revision  of  the  Evaluation  Design  Plan.  The  steps  resulted  in  an  iterative  draft- 
review-revise  process  that  ensured  important  issues  were  identified  and  resolved  in  progressive 
fashion.  The  review  process  played  a  critical  role  in  producing  a  defensible  planning  foundation 
for  the  evaluation.  In  the  final  step,  the  design  parameters  and  sampling  strategy  were  adjusted 
to  account  for  the  expected  availability  of  resources  to  increase  the  feasibility  of  successfully 
implementing  the  evaluation. 

As  the  Evaluation  Design  Plan  reached  a  reasonable  stage  of  maturity,  the  team  initiated 
work  on  the  Data  Collection  and  Management  Plan.  This  process  began  with  the  preparation  of 
a  blueprint  for  the  document  in  the  form  of  an  annotated  outline.  The  team  then  composed  the 
requisite  sections  by  building  on  and  elaborating  the  relevant  parameters  established  in  the 
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Evaluation  Design  Plan.  Each  step  required  detailed  analysis,  and  certain  sections  (e.g.,  detailed 
scheduling,  database  requirements)  involved  creating  new  planning  information.  Throughout  the 
entire  process  it  was  necessary  to  resolve  numerous  questions  and  issues.  Substantial  effort  went 
into  building  and  refining  scheduling  and  staffing  parameters  for  the  complex  family  of  data 
collection  activities.  These  parameters  were  important  because  of  their  key  role  in  shaping  the 
resource  requirements  for  the  actual  data  collection  to  be  conducted  in  follow-on  efforts.  Here, 
too,  the  team  employed  an  iterative  draft-review-revise  process.  At  key  points  they  cross-walked 
the  draft  Data  Collection  and  Management  Plan  against  the  latest  version  of  the  Evaluation 
Design  Plan  to  ensure  accurate  and  consistent  linkages. 

Development  of  Data  Collection  Instruments 

To  enable  the  collection  of  comprehensive  data  on  the  impact  of  Infantry  ALC  training, 
the  team  designed  a  family  of  data  collection  tools  and  developed  a  portion  of  them.  The  goal 
was  to  establish  standard  tools  tailored  to  the  specific  sources  of  data,  relying  on  a  mixture  of 
objective,  self  report,  and  subjective  measurement  techniques.  The  effort  built  on  the  Evaluation 
Design  Plan's  measurement  strategy  as  well  as  the  master  list  of  measures  compiled  earlier  as 
the  final  version  of  the  course  impact  metrics. 

To  ensure  a  coherent  measurement  architecture  for  the  data  collection  instruments,  the 
development  effort  began  with  a  front-end  design  stage.  Using  the  Evaluation  Design  Plan's 
inventory  of  required  data  collection  instruments,  the  team  defined  the  design  parameters  for 
each  instrument.  The  parameters  included  target  audience,  presentation  mode  (e.g.,  hardcopy), 
time  limit  for  completing  the  instrument,  item  limit,  response  mode  (e.g.,  rating  scale),  modeling 
mechanism  (e.g.,  example  entries),  and  presentation  approach  (to  include  organization  and 
layout).  The  team  also  listed  the  types  of  data  that  defined  the  measurement  scope  of  each 
instrument.  The  design  parameters  and  data  types  for  all  the  instruments  were  compiled  into  an 
outline  design  document  that  included  underlying  principles  and  special  considerations.  After 
two  rounds  of  internal  review,  this  document  was  finalized  to  provide  the  roadmap  for 
developing  the  data  collection  instruments  themselves. 

With  the  outline  design  document  in  hand,  the  team  selected  two  survey  questionnaires 
(start-of-course  student  survey,  and  end-state  participant  survey)  for  initial  development  as 
exemplars.  For  each  exemplar,  they  used  the  master  list  of  measures  to  identify  specific  data 
elements  for  inclusion.  The  investigators  also  identified  supplemental  data  elements  (e.g.,  parent 
unit  and  demographic  information).  The  team  then  organized  the  mix  of  data  elements  in 
clusters  of  related  items,  where  appropriate,  and  constructed  the  quantitative  (rating-based)  and 
qualitative  items  for  the  instrument.  After  adding  administrative  elements  (e.g.,  date,  location, 
instructions),  the  investigators  obtained  comments  from  internal  reviewers  and  used  the 
comments  to  revise  the  draft  instrument.  A  second  round  of  internal  review  with  different  team 
member(s)  led  to  another  revision  of  the  questionnaire  along  with  a  verification  check  of  its 
completeness  against  the  master  list  of  measures. 

After  the  exemplar  instruments  had  been  through  at  least  two  review-revision  cycles,  the 
team  used  them  to  guide  the  construction  of  the  remaining  instruments  selected  for  development 
in  this  project.  The  same  iterative  develop-review -revise  process  was  followed  during  this  stage. 
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A  final  review  of  editorial  quality  was  performed,  and  the  final  instruments  were  produced  in 
print-ready  digital  files  suitable  for  fonnatting  in  machine-scan  form.  Precision  file  naming 
conventions  were  adopted  to  avoid  confusion  regarding  version  control. 

No  effort  was  made  in  the  current  project  to  pilot  test  any  data  collection  instruments. 
That  important  step  was  reserved  for  the  follow-on  effort,  to  accommodate  later  modifications 
that  might  result  from  near-term  planning  and  coordination  with  supporting  units.  The  team 
included  an  outline  plan  for  the  pilot  test  in  the  Data  Collection  and  Management  Plan. 

Development  of  Data  Collector’s  Guide 

The  final  product  of  the  present  research  was  the  Data  Collector ’s  Guide,  designed  to 
serve  as  a  daily  job  aid.  The  guide  was  meant  to  help  standardize  data  collection  processes  and 
maintain  the  quality  and  integrity  of  the  data.  The  primary  goal  was  to  equip  the  data  collectors 
with  an  easy-to-use  job  aid  containing  only  minimum  essential  information. 

The  team  began  this  effort  by  developing  a  small  set  of  design  principles  for  the  Data 
Collector’s  Guide,  based  on  the  investigators’  understanding  of  the  data  collection  environment. 
Applying  these  design  principles,  team  members  then  selected  essential  information  from  the 
Data  Collection  and  Management  Plan  and  condensed  it  for  concise  presentation.  A  few  unique 
elements  of  information  were  created,  such  as  duties  of  a  data  collector  and  travel  procedures. 
The  team  organized  the  topics  in  a  sequence  designed  to  make  sense  from  a  data  collector’s 
perspective.  Substantial  effort  went  into  constructing  a  family  of  checklists  detailing  steps  and 
timelines  for  each  primary  duty  (activity)  of  a  data  collector.  This  typically  involved  creating 
new  procedural  details. 

After  integrating  the  various  components  into  a  unified  package,  the  team  cross-walked 
the  draft  Data  Collector ’s  Guide  against  the  Evaluation  Design  Plan  and  the  Data  Collection 
and  Management  Plan  to  ensure  consistency  across  documents.  The  investigators  then 
orchestrated  two  rounds  of  internal  review  followed  by  revision  of  the  guide.  Finalization  of  the 
guide  included  a  careful  review  of  editorial  quality.  Beyond  review  by  SMEs,  no  effort  was 
made  in  the  current  project  to  pilot  test  the  Data  Collector ’s  Guide.  That  critical  step  was 
targeted  for  the  follow-on  effort,  to  ensure  realistic  testing  in  an  operational  environment. 

Products  of  the  Research 

This  section  describes  and  characterizes  the  products  created  to  serve  as  enablers  for  the 
planned  evaluation.  The  presentation  is  organized  as  follows: 

•  Course  Impact  Metrics 

•  Evaluation  Design  Plan 

•  Data  Collection  and  Management  Plan 

•  Data  Collection  Instruments 

•  Data  Collector’s  Guide 

The  family  of  methodological  documents  produced  in  this  project  forms  a  comprehensive 
foundation  for  follow-on  work  to  coordinate  and  execute  the  training  transfer  evaluation  of  the 
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revised  Infantry  ALC  Phase  II  POI.  Presented  intact  in  the  appendixes  of  this  report,  the 
documents  provide  building  blocks  for  a  future  research  team  to  harness  in  their  efforts  to  collect 
meaningful  data.  The  documents  will  help  investigators  standardize  data  collection  procedures 
across  field  teams,  units,  and  installations. 

Course  Impact  Metrics 

A  series  of  products  resulted  from  the  phased  efforts  to  develop  metrics  for  measuring 
Infantry  ALC’s  operational  impact.  These  products  will  organize  the  discussion  in  this 
subsection,  as  follows: 

•  Behavioral  outcomes 

•  Behavioral  taxonomy 

•  Competence  indicators 

•  Measurement  categories 

•  Final  list  of  measures  with  data  elements 

As  the  cornerstone  of  the  metrics  development  process,  the  list  of  behavioral  outcomes 
represented  the  implied  learning  outcomes  of  the  ALC  program,  rather  than  the  terminal  learning 
objectives.  Rooted  in  the  POI,  the  behavioral  outcomes  were  organized  according  to  the  POI’s 
modules  and  subjects.  Under  the  four  modules  were  twenty  subjects  that  spawned  a  total  of  268 
behavioral  outcomes.  For  example,  the  behavioral  outcomes  included  (a)  NCO  coaches  Soldiers 
in  land  navigation  procedures  and  (b)  NCO  seeks  mentoring  from  Platoon  Sergeant.  Many  of  the 
outcomes  were  common  across  multiple  subjects  (e.g.,  coaching  Soldiers),  differentiated  only  by 
domain  specifics.  The  complete  list  of  behavioral  outcomes  is  presented  in  Appendix  B. 

The  list  of  behavioral  outcomes  gave  rise  to  the  Behavioral  Taxonomy  that  offered  a 
systematic  schema  for  organizing  ALC-related  learning  outcomes.  As  Appendix  C  shows,  the 
taxonomy  hinged  on  three  areas — warfighting  proficiency,  leadership,  and  training — at  the  top 
level,  with  14  categories  under  these  areas  (e.g.,  training  management,  tactical  proficiency). 
Under  each  category  were  3-7  representative  behaviors,  such  as  performing  combative  skills  to 
standard  and  giving  proper  commands  to  Soldiers.  A  prime  advantage  of  the  taxonomy  was  its 
independence  from  the  structure  of  the  ALC  POI,  which  gave  it  a  useful  measure  of  generality 
and  comprehensiveness. 

The  search  of  Army  publications  for  doctrinally  anchored  NCO  competencies  produced 
the  third  product  in  the  metrics  series — a  list  of  26  competence  indicators  intended  to  help  keep 
the  metrics  focused  on  parameters  of  operational  significance.  Because  of  their  doctrinal 
connection,  the  indicators  reflected  the  job  performance  expectations  for  NCO-squad  leaders 
working  in  tactical  units.  As  seen  in  Appendix  D,  the  competencies  were  organized  under  three 
broad  categories: 

•  Leadership  behavior  (7  indicators  such  as  guiding  Soldiers  and  fostering  teamwork) 

•  Job  performance  (14  indicators  such  as  communicating  and  demonstrating  competence) 

•  Unit  readiness  (5  indicators  such  as  maintaining  and  reporting  equipment  readiness) 

The  initial  list  of  metrics  integrated  features  of  the  behavioral  taxonomy  and  the  NCO 
competencies.  After  internal  review  and  revision,  the  list  contained  57  specific  measures 
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organized  within  five  functional  categories.  The  number  of  measures  in  each  category  ranged 
from  6  to  2 1 .  A  cross-check  verified  that  the  revised  list  addressed  all  aspects  of  the  behavioral 
taxonomy  and  the  inventory  of  NCO  competencies.  The  metrics  categories  included: 

•  Warfighting  proficiency  (e.g.,  operating  weapons,  perfonning  Common  Tasks) 

•  Military  leadership  (e.g.,  troop  leading,  communicating,  developing  subordinates) 

•  Resource  management  (e.g.,  accounting  for  resources,  supervising  maintenance) 

•  Unit  training  (e.g.,  developing  training  plan,  preparing  for  and  managing  events) 

•  Self-development  (e.g.,  working  to  improve  leadership  and  warfighting  skills) 

When  the  research  team  discussed  the  initial  list  of  metrics  with  Infantry  ALC  leaders 
and  instructors  in  small-group  interviews  (1-3  participants  per  group),  the  resulting  feedback  led 
to  substantive  changes  in  the  list’s  organization.  As  a  result  of  recommendations  obtained  in 
interviews,  the  team  adopted  the  five  roles  of  a  multi-functional  NCO  (Infantry  ALC  Course 
Manager,  personal  communication.  May  7,  2008)  as  the  categories  for  organizing  the  measures 
of  training  impact.  Table  1  lists  the  roles  along  with  the  definitions  developed  for  this  effort.  By 
and  large,  the  multi-functional  roles  subsumed  the  five  categories  of  the  initial  metrics  list,  with 
one  major  exception.  The  role  of  ambassador  brought  a  new  dimension  to  the  metrics  and  led  to 
the  creation  of  new  measures  of  ALC  impact.  The  feedback  from  the  ALC  leaders  and 
instructors  also  led  to  reducing  the  quantity  of  measures  and  prioritizing  the  final  measures 
within  each  category  and  sub-category. 

Table  1 

The  Five  Roles  of  a  Multi-Functional  NCO 


Role 

What  Does  It  Mean? 

Critical  and 
Creative  Thinker 

The  cognitive  dimensions  related  to  adaptive  performance;  characterized  by 
taking  into  account  unique  conditions,  anticipatory  thinking,  defining  and 
analyzing  problems  or  challenges,  thinking  through  alternatives,  and  learning 
through  reflection. 

Warrior  Leader 

The  behavioral  aspects  of  being  a  competent  warfighter  who  leads  combat  units; 
includes  expertise  in  tactical  tasks  and  functions,  mastery  of  technical  skills,  and 
proficiency  in  military  leadership  and  teamwork. 

Leader 

Developer 

The  team- focused  dimensions  of  developing  and  maintaining  tactically  proficient 
squads/sections/crews;  encompasses  shaping  the  behaviors  of  subordinates, 
serving  as  primary  trainer  and  evaluator,  and  improving  one’s  own  capabilities. 

Ambassador 

The  behavioral  and  attitudinal  dimensions  of  integrating  the  unit  with  non- Army 
elements  including  surrounding  communities;  characterized  by  projecting  a 
professional  image,  conforming  with  local  standards  of  behavior,  collaborating 
externally,  and  leveraging  community  assets  as  mission  enablers. 

Resource 

Manager 

The  behavioral  aspects  of  accounting  for  and  protecting  unit  personnel, 
equipment,  and  supplies;  includes  implementing  accountability  procedures, 
accomplishing  logistics  functions,  monitoring  status  of  resources,  and  exercising 
custodial  responsibility. 

Beyond  realigning  the  measurement  categories,  the  ALC  leaders  and  instructors  offered 
other  ideas  for  improving  the  metrics.  Their  suggestions  led  to  several  modifications: 

•  Selective  addition  of  sub-categories  to  improve  the  organization  of  measures 
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•  Reduction  of  the  volume  of  measures  by  deleting  overlapping  items  and  combining 
closely  related  items 

•  Creation  of  ambassador  measures  that  better  fit  the  category 

•  Ordering  measures  according  to  rated  importance 

•  Integration  and  streamlining  of  measures  under  Warrior  Competencies 

•  Deletion  of  measures  where  a  squad  leader  has  no  control  or  direct  influence 

•  Expansion  of  the  “during  training”  qualifier  to  include  real-world  operations 

The  final  metrics  package  (see  Appendix  E)  contained  44  measures  organized  under  6 
categories — those  listed  in  Table  1  plus  the  global  impact  category  that  accommodated  overall 
measures  and  transfer  climate  factors.  The  structure  of  the  final  package  is  outlined  in  Table  2. 
As  the  table  shows,  three  of  the  six  categories  were  split  into  sub-categories.  The  number  of 
measures  in  a  given  category  or  sub-category  ranged  from  2  to  5,  with  a  median  of  4  measures. 
The  vast  majority  of  the  measures  addressed  the  attributes  or  behavior  of  an  individual  NCO,  but 
a  few  dealt  with  course  impact  at  the  level  of  individual  squad  members  or  a  collective  squad/ 
platoon.  The  complete  metrics  package  provided  the  basis  for  defining  data  elements  that  could 
be  mapped  into  data  collection  instruments  for  the  evaluation. 


Table  2 

Structure  of  Final  Metrics  Package 


Category 

Sub-Category 

Measures 

Critical  &  Creative  Thinker 

None 

5  measures,  individual  (NCO) 

Warrior  Leader 

Military  Leadership 

5  measures,  individual  (NCO,  squad) 

Warrior  Competencies 

3  measures,  individual  (NCO) 

Leader  Developer 

Counseling/Coaching/Mentoring 

3  measures,  individual  (NCO) 

Training  Subordinates 

5  measures,  individual  (NCO) 

Shaping  Unit  Performance 

4  measures,  individual  (squad)  +  unit 

Expanding  Own  Competencies 

4  measures,  individual  (NCO) 

Ambassador 

All  Environments 

4  measures,  individual  (NCO,  squad) 

Operational  Environment 

2  measures,  individual  (NCO) 

Resource  Manager 

None 

4  measures,  individual  (NCO,  squad) 

Global  Impact 

None 

5  measures,  individual  +  unit 

The  final  metrics  package  comprised  the  master  list  of  measures.  To  provide  sufficient 
detail  to  define  measurement  procedures,  the  final  list  took  each  measure  down  to  the  level  of 
data  elements  (see  Appendix  E).  The  data  elements  operationally  defined  the  measures,  pointing 
to  specific  aspects  that  could  feasibly  be  measured.  It  was  these  elements  that  spawned  the 
individual  items  for  questionnaires,  focus  group  protocols,  and  other  data  collection  instruments. 
The  number  of  elements  under  a  given  measure  ranged  from  2  to  6  for  the  primary  measures 
(median  =  4,  mode  =  4)  and  from  4  to  12  for  the  global  measures  (median  =  9).  Table  3  gives 
examples  of  the  data  elements  for  each  measurement  category. 
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Table  3 

Example  Data  Elements,  by  Measurement  Category 


Category 

Example  Data  Elements 

Critical  and 
Creative  Thinker 

•  Ratings  of  extent  to  which  ALC  events  provided  relevant  practice 

•  Examples  of  ALC’s  impact  on  NCO’s  contributions  to  lessons  learned 

•  Comments  re:  ALC’s  impact  on  NCO’s  adaptive  thinking 

Warrior  Leader 

•  Ratings  of  squad’s  improvement  in  teamwork,  morale,  respect,  etc. 

•  Squad’s  adverse  personnel  incidents  (AWOL,  disciplinary  actions,  etc.) 

•  Comments  re:  ALC’s  impact  on  NCO’s  implementation  of  TTP 

Leader 

Developer 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  coaching,  mentoring,  etc. 

•  Listing  of  professional  development  sessions  in  which  NCO  participated 

•  Comments  re:  ALC’s  impact  on  squad’s  performance  of  battle  drills 

Ambassador 

•  Ratings  of  extent  to  which  ALC  enhanced  NCO’s  cultural  awareness 

•  Listing  of  outside-the-unit  activities  or  events  which  NCO  led  or  participated  in 

•  Comments  re:  ALC’s  impact  on  NCO’s  consideration  of  community  factors 

Resource 

Manager 

•  Ratings  of  extent  to  which  ALC  training  addressed  managing  unit  resources 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  resource  management 

•  Comments  re:  ALC  impact  on  NCO  management  of  equipment  maintenance 

Global  Impact 

•  Ratings  of  time  and  opportunities  available  to  utilize  KSAs  learned  in  ALC 

•  Squad  and  platoon  performance  ratings  by  training  center  observer/controllers 

•  Comments  regarding  ALC  benefits  to  NCO,  squad,  platoon,  etc. 

Note.  AWOL  =  absent  without  leave;  TTP  =  tactics,  techniques  and  procedures. 


As  explained  in  the  In  troduction  of  this  report,  the  evaluation  strategy  aimed  to  address 
both  near  and  far  transfer  of  ALC  training,  as  well  as  key  characteristics  of  the  transfer  climate. 
That  the  final  metrics  package  achieved  this  goal  is  seen  in  Table  4,  which  sorts  the  various  types 
of  data  elements  according  to  the  measurement  area  that  each  type  of  element  fits  best.  All  four 
levels  of  Kirkpatrick’s  (1994)  measurement  taxonomy  are  represented,  especially  the  behavior 
and  results  levels. 


Evaluation  Design  Plan 

As  the  prime  planning  document,  the  Evaluation  Design  Plan  (Appendix  F)  specified  the 
general  framework  and  technical  approaches  for  conducting  a  systematic  and  comprehensive 
assessment.  The  plan  established  the  foundation  on  which  to  construct  the  detailed  planning 
documents  and  the  data  collection  procedures.  By  incorporating  the  deliberate  limitations  as 
well  as  the  expected  resource  constraints,  the  document  bypassed  the  ideal  or  optimal  evaluation 
design  in  favor  of  a  realistic  plan  offering  a  high  probability  of  implementation  success. 

As  seen  in  Appendix  F,  the  Evaluation  Design  Plan  was  organized  into  eleven  elements: 
background,  goals  and  objectives,  design,  technical  approach,  sampling  strategy,  assumptions, 
measurement  strategy,  measurement  architecture  (metrics  categories),  types  of  measures,  data 
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collection  basics,  and  scheduling  information.  The  goals  of  the  evaluation  revolved  around 
informing  Infantry  ALC  program  decisions  and  helping  tactical  units  realize  greater  payoff  from 
the  program.  The  objectives  focused  on  measuring  improvement  in  the  capabilities  and 
performance  of  individual  NCOs  and  their  units,  along  with  contributing  factors.  The  plan’s 
remaining  elements  outlined  the  major  dimensions  essential  to  accomplishing  the  goals  and 
objectives.  A  briefing  chart  format  was  chosen  to  streamline  the  presentation  of  information  and 
to  later  facilitate  the  process  of  briefing  U.  S.  Army  Forces  Command  (FORSCOM)  stakeholders 
about  the  program. 

Table  4 

Final  Metrics  Package ’s  Coverage  of  Measurement  Areas 


Area 

Data  Element  Types 

Near  Transfer 

•  Perceived  extent  of  ALC  transfer  to  job-relevant  behaviors  and  performance 

•  Perceived  ALC  impact  on  job-relevant  KSAs,  behaviors,  and  attitudes 

•  Perceived  ALC  impact  on  job  performance  style  (e.g.,  efficiency,  productivity) 

•  Perceived  ALC  impact  on  ability  to  improve  own  professional  skills 

•  Perceived  extent  of  positive  feedback  received  from  superiors  and  others 

•  Scores  achieved  in  special  tests  of  tactical  judgment/problem  solving 

•  Comments  re:  ALC  impact  on  job-relevant  KSAs,  behaviors,  and  attitudes 

•  Tracking  or  recall  of  activities/events  of  special  interest 

•  Examples  (positive,  negative)  of  performing  specific  job-relevant  behaviors 

Far  Transfer 

•  Perceived  ALC  impact  on  subordinates’  KSAs,  attitudes,  and  understanding 

•  Perceived  improvement  in  squad  performance,  behaviors,  and  attitudes 

•  Perceived  ALC  impact  on  accomplishment  of  specific  unit  functions/processes 

•  Perceived  ALC  impact  on  specific  aspects  of  mission  accomplishment 

•  Perceived  ALC  impact  on  individual’s  or  squad’s  standing  and  contributions 

•  Perceived  ALC  impact  on  ability  to  improve  unit  training  and  operations 

•  Scores  recorded  for  NCOs/subordinates/squad  in  unit  training  and  testing  events 

•  Comments  re:  ALC  impact  on  subordinate/squad  KSAs,  attitudes,  performance 

•  Comments  re:  ALC-attributed  improvement  in  unit  training  and  operations 

•  Comments  re:  ALC  benefits  to  individual  and  unit 

•  Examples  of  accomplishments  or  contributions  to  specific  unit  functions 

•  Examples  (positive/negative)  of  ALC  impact  on  squad  members’  KSAs 

•  Examples  (positive/negative)  of  ALC  impact  on  squad  functions  and  dynamics 

Opportunity  & 
Climate 

•  Perceived  extent  of  ALC  training  or  practice  on  specific  KSAs  and  behaviors 

•  Perceived  relevance  of  ALC  training  to  job  performance  and  behaviors 

•  Perceived  extent  to  which  a  unit  values/encourages  application  of  ALC  learning 

•  Tracking  of  job-related  activities  involving  application  of  ALC  learning 

•  Retrospective  estimates  of  time  available  to  apply  ALC  learning  on  the  job 

•  Retrospective  ratings  of  opportunities  to  apply  ALC  learning  on  the  job 

•  Retrospective  ratings  of  factors  hindering  application  of  ALC  learning 

•  Recall  of  conditions  or  events  that  could  prompt  specific  NCO  behaviors 

Note.  KSAs  =  knowledge,  skills,  and  attributes. 
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Embedded  in  the  Evaluation  Design  Plan  were  specifications  resulting  from  decisions 
about  the  scope  of  the  planned  effort.  Table  5  lists  the  major  dimensions  defining  the  scope  and 
summarizes  the  provisions  specified  in  the  plan.  Two  of  the  more  important  delimiters  dealt 
with  which  Army  components  and  what  geographic  domain  to  target  in  the  evaluation.  The 
decision  to  include  only  Active  Component  units  limited  the  ability  to  generalize  the  findings  of 
the  evaluation,  as  did  the  decision  to  restrict  the  geographic  scope  to  the  Continental  United 
States  (CONUS).  However,  these  limitations  were  driven  by  practical  reasons  related  primarily 
to  deployment  realities  and  resource  constraints — reasons  that  were  considered  compelling. 

Table  5 

Scope  of  Evaluation 


Dimension 

What’s  Planned? 

Purpose 

Determination  of  operational  impact  of  the  new  1  IB  ALC  Phase  II  POI 

Stakeholders 

Army  Infantry  School,  NCOA-Benning,  FORSCOM  Commanders 

Army  Components 

Active  Component  units  only,  preferably  in  reset/train  stage 

Geographic  Domain 

Continental  United  States  only  (force  projection  base) 

Measurement  Scope 

Focus  on  transfer  of  ALC  learning  to  job  performance  and  unit  readiness 

Calendar  Scope 

Ten- month  window  anchored  to  selected  schoolhouse  and  MTT  courses 

Primary  Outcome 

Empirical  data  to  support  programmatic  decisions  regarding  the  ALC  POI 

Note.  MTT  =  mobile  training  team. 


Within  the  constraints  of  the  evaluation’s  scope,  the  technical  approach  was  constructed 
in  the  form  of  the  primary  evaluation  design  parameters  listed  in  Table  6.  A  simple  research 
paradigm  was  adopted  in  which  a  framework  of  putative  change  and  multi-source  assessment 
replaced  consideration  of  including  a  control  group.  Arguing  against  a  control  group  were 
methodological  issues  (e.g.,  unavailability  of  a  “control”  POI  during  the  evaluation  period)  and 
resource  constraints.  For  the  heart  of  the  sampling  approach,  it  was  decided  to  anchor  the 
eligible  NCOs  to  ALC  courses  rather  than  units  or  installations.  Anchoring  to  scheduled  courses 
would  optimize  sample  sizes  and  facilitate  the  gathering  of  data  while  NCOs  were  ALC  students. 
Specifying  a  minimum  application  period  during  which  NCOs  could  employ  what  they  learned 
from  ALC  was  recognized  as  an  essential  part  of  the  technical  approach. 

Several  of  the  parameters  in  Table  6  comprised  the  tenns  of  the  measurement  strategy. 
The  six  metrics  categories  discussed  earlier  (see  Table  3)  provided  the  framework  for  structuring 
the  measures  of  Infantry  ALC  impact,  with  an  overarching  emphasis  on  linking  measures  to  unit 
training  and  operations.  The  strategy  called  for  measuring  individual  and  collective  impact 
parameters  as  well  as  opportunity  parameters  related  to  organizational  programs — both  the 
Infantry  ALC  POI  and  unit  training/operations.  The  mix  of  data  sources  and  measurement 
methods  led  to  the  specification  of  four  types  of  data  collection  techniques — questionnaires, 
focus  groups,  situational  judgment  tests  (SJTs),  and  records  mining.  The  decision  to  collect  data 
in  a  supervised  setting  (vs.  a  self-administered  setting)  resulted  from  a  strong  desire  to  maximize 
participation  rates  and  avoid  lost  data. 
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Table  6 

Primary  Evaluation  Design  Parameters 


Parameter 

Summary 

Hypothesis 

New  Phase  II  POI  will  improve  1  IB  ALC’s  impact  among  tactical  units 

Research  Paradigm 

Multi-source  assessment  in  putative  change  framework  (no  control  group) 

Sampling  Approach 

Target  early  courses  conducted  with  new  POI  to  define  sampling  domain 

Application  Period 

At  least  2  months  as  a  post-ALC  squad  leader,  5-6  months  desired 

Variables  of  Interest 

Type  of  delivery  (schoolhouse  vs.  MTT),  installation,  demographic  factors 

Measures  Framework 

Six  categories  (based  on  multi-functional  NCO)  from  metrics  package 

Measurement  Levels 

Individual  (NCOs,  squad  members),  collective  (squad),  POI  (opportunity) 

Data  Sources 

NCO-participants,  in-the-know  associates  (leaders  and  peers),  unit  records 

Measurement  Methods 

Mix  of  objective,  self-report,  self-rating,  and  other-rating  techniques 

Data  Collection  Tools 

Questionnaires,  focus  groups,  situational  judgment  tests,  records  mining 

Measurement  Setting 

Data  collection  supervised  by  ARI  researchers  in  operational  environment 

The  principal  elements  of  the  sampling  strategy  for  the  evaluation  appear  in  Table  7.  To 
define  the  sampling  pool  within  the  target  population  of  new  POI  graduates,  the  strategy  targeted 
five  early  courses  using  the  new  POI — two  schoolhouse  courses  and  three  mobile  training  team 
(MTT)  courses  (an  MTT  course  hosts  fewer  students  than  a  schoolhouse  course).  It  was 
reasoned  that  bypassing  the  first  few  courses  to  utilize  the  new  POI  would  allow  the  modified 
instructional  process  to  stabilize.  To  maximize  the  sample  sizes,  the  plan  called  for  enrolling  all 
eligible  students  in  the  selected  courses  as  participants.  Because  of  constraints  expected  in 
evaluation  funding,  only  six  of  the  eight  eligible  CONUS  installations  were  targeted,  three  of 
which  would  house  only  schoolhouse  participants. 

Table  7 

Elements  of  Sampling  Strategy 


Element 

Summary 

Target  Population 

Graduates  of  new  POI  ( participants )  plus  associates  in  parent  units 

Course  Targeting 

2  schoolhouse  and  3  MTT  courses  that  follow  initial  iterations  of  new  POI 

Geographic  Targeting 

Six  (of  eight)  CONUS  installations  where  1  IB  ALC  graduates  work 

Parent  Unit  Types 

Aim  for  two  light,  two  heavy,  and  two  Stryker  BCTs  across  installations 

NCO  Participants 

All  graduates  of  selected  courses  assigned  to  units  2+  mos  from  deploying 

Unit  Associates 

Leaders  and  peers  who  observe  a  participant  pre-  and  post-ALC 

Selection  Process 

Based  on  criteria — duty  position,  CONUS  stationing,  temporal  parameters 

Sample  Sizing 

Determined  by  number  of  courses  and  criteria  (no  minimum  or  maximum) 

Expected  Attrition 

10-20%  of  MTT  and  schoolhouse  participants,  up  to  25%  of  associates 

Participation  Surety 

Reliance  on  command  emphasis  coordinated  through  command  channels 

Note.  BCT  =  Brigade  Combat  Team. 
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To  sample  different  types  of  Brigade  Combat  Teams  (BCTs),  the  strategy  aimed  to 
sample  light,  heavy,  and  Stryker  BCTs — to  the  extent  possible — for  the  schoolhouse  and  MTT 
cohorts.  Applying  the  criteria-based  process  for  selecting  NCO -participants  and  their  informed 
associates  was  expected  to  yield  approximately  500  participants  (evenly  divided  between 
schoolhouse  and  MTT  courses)  and  approximately  1000  associates  (2  per  participant). 

The  Evaluation  Design  Plan's  discussion- free  fonnat  (briefing  charts)  maintained  the 
focus  on  central  parameters  and  issues,  and  it  minimized  the  effort  required  to  revise  the  plan 
during  the  multiple  cycles  of  review  followed  by  revision.  However,  the  lean  presentation  style 
largely  ignored  the  options  rejected  for  key  parameters  and  the  rationale  behind  the  final 
decisions.  The  following  bullets  outline  major  options  and  the  reason(s)  for  rejecting  them: 

•  Exclusion  of  near  transfer — excluding  near  transfer  would  limit  the  knowledge  base 
appreciably  and  hinder  interpretation  of  far  transfer  effects. 

•  Inclusion  of  control  group — complete  data  collection  would  be  impossible  with  a 
control  group  and  the  pre-post  comparison  paradigm  should  suffice. 

•  Including  subordinates  as  data  sources — available  resources  would  not  support  the 
additional  data  collection  efforts. 

•  Inclusion  of  associates  with  post-ALC  acquaintanceship  only — such  a  provision  would 
negate  the  evaluation’s  putative  change  paradigm. 

•  Comparison  between  units  at  different  life  cycle  stages — such  comparison  would 
undermine  the  goal  of  5-6  months  for  the  application  period. 

•  Sample  size  target  as  low  as  200  participants — larger  sample  size  would  enhance 
confidence  in  findings  without  increasing  costs  unacceptably. 

•  Sampling  up  to  4  associates  (vs.  2)  per  participant — increased  costs  would  outweigh 
the  statistical  benefits  of  larger  sample  sizes. 

•  Desired  application  period  of  only  2-3  months — robust  application  of  ALC  learning 
would  require  5-6  months  in  light  of  today’s  deployment  pressures. 

•  Including  structured  interviews  in  data  collection — the  resource-intensive  technique 
was  abandoned  in  favor  of  higher  payoff  focus  groups. 

•  Omitting  objective  competence  measures  such  as  SJTs — measuring  competencies  by 
subjective  means  only  would  limit  the  credibility  of  the  findings. 

•  De-emphasis  of  self-rating  data — risk  of  inflating  own  capabilities  was  minimized  by 
placing  self-ratings  in  the  context  of  ALC  impact. 

Data  Collection  and  Management  Plan 

The  Data  Collection  and  Management  Plan  (Appendix  G)  provided  the  roadmap  for 
creating  an  ALC  impact  database  of  high  quality,  stopping  short  of  data  analysis.  This  plan  built 
on  the  Evaluation  Design  Plan,  especially  the  measurement  and  sampling  strategies,  and  on  the 
impact  metrics  to  specify  the  procedures  for  collecting  and  managing  robust  data.  With  its  focus 
on  specifications  and  procedures,  this  document  formed  a  detailed  planning  tool  to  guide 
preparation  and  execution  of  data  collection  and  processing  activities  during  the  evaluation.  The 
plan  incorporated  flexibility  to  rescale  specific  features  in  response  to  resource  constraints  and/or 
troop  support  limitations. 
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As  Appendix  G  shows,  the  Data  Collection  and  Management  Plan  described  in  detail  the 
sampling  procedures,  data  requirements,  data  collection  instruments  and  procedures,  scheduling 
parameters,  data  management  procedures,  database  requirements,  resources,  data  collection 
locations,  and  pilot  testing.  Driven  by  the  sampling  goals  established  in  the  Evaluation  Design 
Plan,  the  sampling  procedures  detailed  the  process  and  criteria  for  selecting  participants  and 
associates  from  the  targeted  Infantry  ALC  classes.  The  list  of  data  collection  instruments 
emphasized  survey  questionnaires  and  focus  group  protocols,  but  included  other  instruments 
such  as  SJTs  and  an  application  checklist.  This  section  also  specified  the  process  for  approving 
and  producing  the  instruments,  stressing  standardization  and  quality  control.  The  section  on  data 
collection  procedures  presented  general  guidelines  plus  the  major  requirements — administration, 
advance  preparation,  actions  during  and  after  the  session,  etc. — for  each  type  of  activity  (e.g., 

SJT  sessions).  The  data  management  section  provided  procedural  guidelines  for  identifying  and 
inventorying  data,  transferring  hardcopy  data  into  digital  form,  maintaining  physical  security, 
and  archiving  data.  Several  of  the  sections — data  collection  schedule,  resources,  and  data 
collection  locations — were  designed  to  support  resource  planning  for  the  future  evaluation.  To 
optimize  the  quantity  and  quality  of  actual  data,  the  plan  called  for  data  collection  teams  visiting 
the  schoolhouse  and/or  home  stations  where  the  participants  and  their  associates  would  reside. 

Within  the  Resources  section,  the  staffing  plan  called  for  a  minimum  essential  mix  of 
research  personnel  with  clearly  defined  responsibilities.  The  research  team  included  an 
evaluation  manager,  a  data  manager,  data  collectors  (SMEs  and  generalists)  organized  in  two- 
person  teams,  a  database  technician,  and  a  data  entry  clerk.  No  duties  were  allocated  to  MTT, 
schoolhouse,  or  unit  personnel  beyond  availability  of  an  administrative  point  of  contact  (POC). 

At  a  global  level,  the  framework  for  scheduling  data  collection  activities  entailed  four 
phases  (Table  8)  that  would  take  a  given  group  of  participants  from  the  start  of  their  course 
attendance  through  the  measurement  of  ALC  impact  after  the  prescribed  application  period.  The 
first  three  phases  involved  collecting  data  from  participants  only,  while  the  fourth  phase  (end 
state  assessment)  involved  both  participants  and  associates.  From  start  to  finish,  the  ideal 
timeline  for  data  collection  associated  with  a  given  course  would  extend  across  7-8  months.  If 
unit  circumstances  were  to  shorten  or  lengthen  the  application  period,  the  actual  timeline  could 
range  from  a  minimum  of  3  months  to  something  exceeding  8  months. 

Table  8 

Phases  of  Data  Collection 


Phase 

Desired  Timeframe 

Focus 

Start-of-Course  Snapshot 

First  2  days  of  course 

Demographics,  pre-training  status 

End-of-Course  Snapshot 

Final  3  days  of  course 

Post-training  status  and  reactions 

Application/Incubation 

5-6  consecutive  months 

Application  of  ALC  learning  on  the  job 

End  State  Assessment 

5-6  months  post- ALC 

Multi-source  measurement  of  ALC  impact 

For  detailed  scheduling  of  specific  activities,  the  Data  Collection  and  Management  Plan 
addressed  the  following  parameters:  when,  duration,  cohort  size,  working  group  size,  number  of 
groups,  and  data  collector  demand.  Two  of  these  parameters  (when  and  duration)  had  a  direct 
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influence  on  timing.  The  remaining  parameters  (e.g.,  working  group  size,  number  of  groups) 
exerted  an  indirect  influence  by  determining  the  number  of  iterations  for  a  specific  activity.  The 
final  parameter — data  collector  demand — will  become  important  when  allocating  the  time  of  a 
limited  pool  of  data  collectors. 

A  number  of  issues  arose  while  developing  the  Data  Collection  and  Management  Plan. 
Generally,  the  issues  were  resolved  by  making  reasoned  assumptions,  with  the  understanding 
that  invalid  assumptions  may  necessitate  revising  the  plan  when  data  collection  is  about  to  start. 
The  primary  assumptions  included: 

•  The  proportion  of  Infantry  ALC  students  meeting  the  participant  criteria  will  be  at 
least  70%  for  schoolhouse  courses  and  up  to  95%  for  MTT  courses. 

•  Unit  associates  who  know  each  participant  before  and  after  ALC  attendance  will  be 
available. 

•  Attrition  will  not  exceed  20%  for  participants  and  25%  for  associates. 

•  Course  managers  and  units  will  fully  support  by-name  scheduling  of  personnel. 

•  Battalions  would  strongly  prefer  to  schedule  all  end  state  activities  for  their  personnel 
within  a  one-week  window. 

•  Centralized  production  of  forms  would  preclude  quality  control  problems. 

•  Online  versions  of  data  collection  instruments  will  be  unnecessary. 

•  Units  will  support  providing  data  elements  from  unit  records. 

•  The  data  manager  will  be  capable  of  designing  and  managing  the  central  database. 

•  Installation  facilities  used  by  data  collectors  will  afford  protection  of  stored  data. 

The  Data  Collection  and  Management  Plan  presented  in  Appendix  G  took  into  account 
the  planning  factors  known  during  the  current  project.  It  will  be  imperative  to  update  and  refine 
the  plan  in  the  early  stages  of  the  follow-on  project. 

Data  Collection  Instruments 

The  family  of  data  collection  instruments  stemmed  from  the  applicable  evaluation  design 
parameters  (see  Table  6),  especially  the  data  sources  and  measurement  methods/tools.  Table  9 
lists  the  complete  set  of  instruments,  along  with  the  administration  timeframe  and  measurement 
focus.  As  the  table  shows,  five  types  of  instruments  were  planned:  survey  questionnaires,  focus 
group  protocols,  SJTs,  application  checklist,  and  unit  records-based  capture  form. 

Across  the  entire  family  of  tools,  a  variety  of  data  collection  techniques  was  planned: 
scale-based  ratings,  skills/knowledge  testing,  comments  (both  written  and  oral),  and  activity 
tracking  (recurring  checklist).  In  addition,  mining  of  focus  group  products  and  unit  records  was 
incorporated  selectively.  This  combination  of  techniques  was  designed  to  (a)  satisfy  the  data 
requirements  of  the  impact  metrics,  (b)  accommodate  target  audience  habits  and  preferences,  and 
(c)  diversify  data  collection  efforts.  The  mix  of  techniques  was  tailored  to  each  source  of  data. 

The  outline  design  document  for  the  planned  data  collection  instruments  (Appendix  H) 
provided  the  blueprint  for  constructing  the  actual  tools  and  fonns.  For  each  instrument  identified 
in  Table  9,  the  document  outlined  the  design  parameters  and  measurement  scope.  It  also 
included  notes  about  the  underlying  principles  and  special  considerations  involved  in  building 
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the  instruments.  The  design  guidelines  were  crafted  to  ensure  that  the  family  of  instruments 
would  (a)  exhibit  a  consistent  appearance,  (b)  be  logically  organized  and  easy  to  use,  (c)  fit  the 
constraints  of  the  data  collection  environment,  and  (d)  produce  high-quality  data.  The  design 
guidelines  addressed  the  following  dimensions: 

•  Target  audience  (data  source) 

•  Delivery  medium  (e.g.,  hardcopy  fonns,  group  facilitation) 

•  Practical  limits  (e.g.,  completion  time,  number  of  items) 

•  Prompting  mechanism  (e.g.,  itemized  list  of  response  options) 

•  Response  modes  (e.g.,  rating  scale,  write-in  comments) 

•  Clustering  principle  (e.g.,  grouping  of  related  items) 

•  Special  formatting  (e.g.,  structure  of  SJT  questions) 

•  Modeling  mechanism  (e.g.,  filled-in  example) 

•  Administration  mechanism  (e.g.,  two-person  focus  group  team) 

•  Nature  of  instructions  (e.g.,  geared  for  respondents  or  facilitators) 

•  Test  conditions  such  as  proctoring  (SJTs  only) 

•  Scoring  approach  (SJTs  only) 


Table  9 

Data  Collection  Instruments  Planned  for  the  Evaluation 


Instrument 

When 

Measurement  Focus 

Survey  Questionnaires 

Biographical  Inventory 

Start  of  course 

Demographics,  schooling,  assignments,  deployments 

ALC  Students  Survey  #1 

Start  of  course 

NCO  competencies,  motivation,  course  expectations 

ALC  Students  Survey  #2 

End  of  course 

Opportunities,  NCO  competencies,  learning  outcomes 

End  State  Participant  Survey 

End  state 

Change  in  own  and  unit  behavior  and  performance 

Unit  Leaders  Survey 

End  state 

Change  in  NCO  and  unit  behavior  and  performance 

Unit  Peers  Survey 

End  state 

Change  in  NCO  and  unit  behavior  and  performance 

Focus  Group  Protocols 

ALC  Students  Focus  Group 

End  of  course 

Opportunities,  learning  outcomes,  expectations 

Participants  Focus  Group 

End  state 

Behavior/perfonnance  changes,  examples,  benefits 

Unit  Leaders  Focus  Group 

End  state 

Behavior/perfonnance  changes,  examples,  benefits 

Unit  Peers  Focus  Group 

End  state 

Behavior/perfonnance  changes,  examples,  benefits 

Miscellaneous  Instruments 

Situational  Judgment  Test  #1 

Start  of  course 

Objective  NCO  abilities  (pre-coursework) 

Situational  Judgment  Test  #2 

End  of  course 

Objective  NCO  abilities  (post-coursework) 

Application  Checklist 

Recurring 

On-the-job  application  opportunities,  activity  examples 

Unit  Records  Capture  Form 

End  state 

Personnel  incidents,  individual/unit  proficiency  scores 

For  the  sake  of  a  consistent  quantitative  response  framework,  an  “agree-disagree”  rating 
scale  served  as  the  predominant  scaling  tool  in  questionnaires.  Omission  of  a  neutral  point  (“no 
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opinion”)  discouraged  fence-sitting,  but  a  “not  performed”  or  “not  observed”  option  was  used 
routinely.  Alternative  scales  (e.g.,  confidence  in  own  ability)  were  used  when  warranted. 

As  discussed  earlier,  one  of  the  evaluation  design  parameters  called  for  collecting  data  in 
the  schoolhouse  or  unit  environment  under  the  supervision  of  researchers  (see  Table  6).  This 
defined  the  global  conditions  for  implementing  the  data  collection  instruments.  Specific  aspects 
of  the  implementation  environment  for  each  type  of  activity  included  the  following: 

•  Survey  questionnaires — large  or  small  groups  of  Soldiers  working  independently,  with 
an  administrator 

•  Focus  groups — small  groups  of  Soldiers  brainstorming  collaboratively  under  the 
guidance  of  a  facilitator,  with  a  separate  note  taker 

•  Situational  judgment  tests — large  groups  of  participants  working  independently, 
monitored  by  a  proctor 

•  Application  checklists — individual  participants  working  alone  at  the  end  of  a  pay 
period,  with  reminders  from  a  unit  action  officer 

•  Unit  records-based  capture — a  researcher  visiting  the  appropriate  headquarters 
elements,  with  assistance  from  a  unit  POC 

Data  Collector’s  Guide 

The  Data  Collector’s  Guide  (Appendix  I)  formed  the  primary  job  aid  for  research  team 
members  who  would  be  collecting  data  in  the  future  evaluation.  Building  on  the  Data  Collection 
and  Management  Plan,  the  guide  packaged  critical  background  and  instructions  for  collecting 
and  protecting  high  quality  data.  With  its  focus  on  who-what-when-where-how  information,  the 
guide  provided  a  highly  practical  tool  to  structure  the  activities  of  data  collectors  as  they  execute 
their  data  collection  and  processing  duties.  The  document  constituted  a  critical  resource  for 
standardizing  the  pivotal  processes  for  gathering  and  assembling  Infantry  ALC  impact  data. 

The  results  of  the  design  stage  for  the  Data  Collector ’s  Guide  are  summarized  in  Table 
10.  The  overall  goal  was  to  produce  a  concise,  all-in-one  job  aid  containing  minimum  essential 
information.  The  decision  to  emphasize  how-to  checklists  backed  up  by  sample  products  came 
from  a  desire  to  arm  a  data  collector  with  simple  procedural  templates.  The  design  goals  set  the 
stage  for  a  guide  that  (a)  focused  on  working  procedures,  (b)  avoided  non-essential  information, 
(c)  could  be  used  anywhere,  and  (d)  would  readily  prove  its  value.  The  goals  shaped  the  form, 
organization,  and  contents  of  the  guide  itself. 

Six  elements  provided  a  streamlined  organizational  scheme  for  the  Data  Collector ’s 
Guide — cover  sheet,  about-this-guide  capsule,  main  body,  project  infonnation,  list  of  data 
collection  activities,  and  procedural  checklists.  Within  the  main  body,  twelve  brief  sections 
introduced  the  researcher  to  the  purpose,  players,  locations,  sources,  phases,  and  tools  of  the  data 
collection  effort.  Two  sections  stated  the  duties  of  a  data  collector  along  with  key  do’s  and 
don’ts.  Also  addressed  were  topics  such  as  dress  code,  travel  requirements,  and  optional  sources 
of  information.  Concluding  the  guide  were  the  all-important  procedural  checklists — one  for  each 
major  activity  to  be  perfonned  by  data  collectors  (e.g.,  scheduling  end  state  sessions, 
administering  an  SJT  session,  submitting  data).  The  compact  size  of  the  guide  (15  pages 
including  the  cover  sheet)  made  it  easy  to  carry  the  document  to  any  work  location. 
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Table  10 

Design  Goals  for  Data  Collector ’s  Guide 


Dimension 

Design  Goal 

Purpose 

Equip  data  collectors  with  easy-to-use,  all-in-one  job  aid 

Scope 

Limit  contents  to  minimum  essential  information  needed  in  the  field 

Range  of  Topics 

Cover  process  overview  plus  procedures  for  major  data  collection  activities 

Organization 

Employ  simple,  functional  scheme  to  support  easy  location  of  information 

Contents 

Combine  main  body  with  step-by-step  checklists  and  sample  products 

Presentation  Mode 

Use  concise  bullets  to  optimize  ease  of  use  and  compactness 

Utilization  Mode 

Gear  document  for  hardcopy  utilization  to  support  unrestricted  mobility 

Self-Sufficiency 

Ensure  stand-alone  document  that  points  to  optional  sources  of  information 

The  Data  Collector ’s  Guide  in  Appendix  I  conveyed  the  essential  information  as  it 
existed  during  the  current  project.  It  will  be  imperative  to  update  and  refine  the  guide  in  the 
early  stages  of  the  follow-on  project,  then  again  after  pilot  testing. 

Recommendations 

The  products  of  the  current  project  fonn  the  foundation  for  the  evaluation  of  Infantry 
ALC’s  operational  impact.  The  Evaluation  Design  Plan  establishes  the  architecture  and 
blueprint  for  conducting  a  systematic  assessment.  The  Data  Collection  and  Management  Plan 
contains  the  roadmap  for  creating  a  high  quality  database  on  ALC  impact.  The  outline  design  for 
the  data  collection  instruments,  in  combination  with  the  metrics  package,  will  guide  the 
completion  of  the  tools  for  capturing  the  required  data  elements.  The  Data  Collector ’s  Guide 
comprises  a  critical  job  aid  for  standardizing  the  data  collection  processes.  In  the  hands  of  the 
follow-on  team,  this  family  of  products  will  enable  rapid  preparation  and  standardized  execution 
of  the  evaluation  activities. 

The  products  contained  in  the  appendixes  of  this  report  account  for  the  planning  factors 
known  during  the  current  project.  Certain  assumptions  may  prove  invalid,  and  new  issues  will 
surely  arise  during  the  detailed  coordination  with  the  Infantry  ALC  community  and  tactical  units. 
It  will  be  imperative  to  update  and  refine  the  products  as  detailed  preparation  for  the  evaluation 
proceeds.  Pilot  testing  will  drive  additional  refinements  as  the  products  are  finalized.  The 
follow-on  team  should  carefully  manage  the  updating  and  refinement  process  to  ensure  that  the 
execution  activities  fully  meet  the  evaluation  objectives. 

During  the  interviews  and  focus  groups  that  reviewed  the  draft  metrics  package,  the  ALC 
stakeholders  offered  suggestions  for  harnessing  the  metrics  of  Infantry  NCO  job  perfonnance. 
Among  the  more  promising  suggestions  were  the  following,  which  may  eventually  apply  to 
career  fields  beyond  Infantry: 

•  Provide  the  metrics  for  optional  use  by  unit  leaders  to  assess  NCO  performance  and/or 
support  counseling  activities. 


22 


•  Employ  the  metrics  as  diagnostics  to  shape  unit  training  and/or  self-development 
efforts  that  focus  on  areas  needing  improvement. 

•  Use  the  metrics  to  modify  or  supplement  the  current  NCO  Evaluation  Report  criteria, 
with  the  goal  of  enhancing  the  professional  development  process. 

•  Create  a  metrics-based  form  that  the  Infantry  School  could  send  to  units  as  a  feedback 
mechanism  to  address  ALC  effectiveness. 

•  Apply  the  metrics  to  develop  a  tool  that  units  could  use  to  provide  recommendations  to 
the  Infantry  School  for  improving  the  ALC  program. 

This  project’s  systematic  planning  products  provide  a  practical  start  point  for  the  unit- 
focused  evaluation  of  the  transfer  of  Infantry  ALC  training.  The  incorporation  of  context 
measures  to  illuminate  why  transfer  effects  do  or  do  not  occur  will  yield  important  clues  about 
definitive  POI  improvements.  As  the  follow-on  team  prepares  for  data  collection,  they  must 
update  and  finalize  the  products,  complete  the  development  of  data  collection  instruments,  and 
conduct  a  thorough  pilot  test.  The  process  should  take  into  account  the  realities  of  the  classroom 
and  unit  environments,  to  include  evolving  interests  of  key  stakeholders  and  emerging  data 
collection  constraints. 

The  future  results  of  the  evaluation  will  shed  light  on  those  aspects  of  the  new  Infantry 
ALC  POI  that  strongly  impact  operational  parameters.  The  findings  will  help  decision  makers, 
course  managers,  and  curriculum  developers  to  refocus  training  so  as  to  enhance  the  transfer  of 
NCO  knowledge  and  skills.  In  turn,  the  stronger  transfer  of  ALC  learning  will  improve  the  job 
performance  of  junior  NCOs  and  strengthen  their  contributions  to  unit  training  and  operations. 
Ultimately,  such  improvements  can  be  expected  to  enhance  combat  readiness  and  force 
effectiveness. 
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APPENDIX  A 


Acronyms  and  Abbreviations 

Note :  This  list  includes  items  from  the  appendixes. 


AAR 

after  action  review 

ABCS 

Army  Battle  Command  System 

ALC 

Advanced  Leaders  Course 

ANCOC 

Advanced  Noncommissioned  Officer  Course 

APFT 

Army  Physical  Fitness  Test 

ARFORGEN 

Army  Force  Generation  (model) 

ARI 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

ARTEP 

Army  Training  and  Evaluation  Program 

AWOL 

absent  without  leave 

BCT 

Brigade  Combat  Team 

BNCOC 

Basic  Noncommissioned  Officer  Course 

CD 

compact  disc 

Cdr 

commander 

Co 

company 

COE 

contemporary  operational  environment 

COIN 

counterinsurgency 

CONUS 

Continental  United  States 

CTC 

Combat  Training  Center 

DC 

data  collection 

DO 

delivery  order 

FarXfer 

Far  Transfer  (project) 

FBCB2 

Force  XXI  Battle  Command  Brigade  and  Below 

FO 

Forward  Observer 

FORSCOM 

U.S.  Army  Forces  Command 

FOUO 

For  Official  Use  Only 

HQ 

headquarters 

IAW 

in  accordance  with 

ID 

identify  /  identification 

ID# 

identification  number 

KSAs 

knowledge,  skills,  and  attributes 

A-l 


METT-TC 

MOS 

MTT 

NCO 

NCOA 

NCOES 

OPORD 

OPTEMPO 

Pit  Ldr 

Pit  Sgt 

POC 

POI 

SFC 

SGT 

SJT 

SME 

SOP 

SSG 

SSN 

T&EO 

TACSOP 

TADSS 

TI 

TIS 

TLP 

TO&E 

TSOP 

UP 

UCMJ 

USR 


mission,  enemy,  terrain  and  weather,  troops  and  support  available,  time 
available,  civil  considerations 

Military  Occupational  Specialty 

Mobile  Training  Team 

noncommissioned  officer 

Noncommissioned  Officer  Academy 

Noncommissioned  Officer  Education  System 

operation  order 

operational  tempo 

Platoon  Leader 

Platoon  Sergeant 

point  of  contact 

program  of  instruction 

Sergeant  First  Class 

Sergeant 

situational  judgment  test 

subject  matter  expert 

standing  operating  procedures 

Staff  Sergeant 

Social  Security  Number 

Training  and  Evaluation  Outline 

tactical  standing  operating  procedures 

training  aids,  devices,  simulators,  and  simulations 

Tactical  Internet 

time  in  service 

troop  leading  procedures 

Table  of  Organization  and  Equipment 

tactical  standing  operating  procedures 

tactics,  techniques,  and  procedures 

Uniform  Code  of  Military  Justice 

Unit  Status  Report 
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APPENDIX  B 


Behavioral  Outcomes 

(Based  on  4-week  ALC  POI  of  April  2008) 


Module 

Subject 

Behavioral  Outcomes 

COMMON 

Land  Navigation 
Testing 

•  NCO  realistically  incorporates  land  navigation  into  training  plans 

•  NCO  explains  land  navigation  procedures  to  Soldiers 

•  NCO  coaches  Soldiers  in  proper  land  navigation  procedures 

•  NCO  answers  Soldiers’  questions  in  accurate  and  timely  fashion 

•  NCO  confirms  use  of  proper  land  navigation  procedures  by  Soldiers 

•  NCO  facilitates  learning  by  tolerating  failure  and  giving  timely  feedback 

•  NCO  evaluates  performance  measures  IAW  Soldier  task  standards 

•  NCO  includes  land  navigation  aspects  in  AAR  discussions 

•  Soldiers  execute  proper  land  navigation  procedures  during  training 

•  Soldiers  perform  to  standard  in  training  exercises 

•  Squads/sections/crews  perform  to  standard  in  training  exercises 

FBCB2 

(Familiarization) 

•  NCO  realistically  incorporates  FBCB2  operations  into  training  plans 

•  NCO  explains  FBCB2  procedures  to  Soldiers  and  fellow  NCOs 

•  NCO  coaches  Soldiers  in  proper  FBCB2  operating  procedures 

•  NCO  answers  Soldiers’  questions  in  accurate  and  timely  fashion 

•  NCO  confirms  proper  setup  and  configuration  of  FBCB2  and  Tl 

•  NCO  facilitates  FBCB2  problem  identification  and  troubleshooting 

•  NCO  identifies  authoritative  sources  of  information  on  FBCB2 

•  NCO  verifies  Soldiers  follow  unit  TACSOP,  adjusts  training  as  needed 

•  NCO  facilitates  learning  by  tolerating  failure  and  giving  timely  feedback 

•  NCO  evaluates  performance  measures  IAW  TACSOP  and  training  plan 

•  NCO  provides  feedback  on  digital  performance  during  AARs 

•  Soldiers  perform  proper  FBCB2  operating  procedures  during  training 

Forward  Observer 
Procedures 

•  NCO  realistically  incorporates  FO  duties  into  training  plans 

•  NCO  explains  FO  procedures  to  Soldiers  and  fellow  NCOs 

•  NCO  coaches  Soldiers  in  proper  FO  procedures 

•  NCO  answers  Soldiers’  questions  in  accurate  and  timely  fashion 

•  NCO  confirms  proper  setup  and  configuration  of  FO  systems 

•  NCO  facilitates  FO  system  problem  identification  and  troubleshooting 

•  NCO  identifies  authoritative  sources  of  information  on  squad  level  FO 

•  NCO  verifies  Soldiers  follow  unit  TACSOP,  adjusts  training  as  needed 

•  NCO  facilitates  learning  by  tolerating  failure  and  giving  timely  feedback 

•  NCO  evaluates  performance  measures  IAW  ARTEP  standards 

•  NCO  provides  feedback  on  FO  performance  during  AARs 

•  Soldiers  execute  proper  FO  procedures  during  training  exercises 

Infantry  Futures 

•  NCO  explains  new  Infantry  initiatives  and  equipment  accurately 

•  NCO  readily  incorporates  new  equipment  into  squad  training  plans 

•  NCO  realistically  incorporates  Infantry  initiatives  into  training  plans 

•  NCO  answers  Soldiers’  questions  in  accurate  and  timely  fashion 

•  NCO  describes  own  efforts  to  stay  abreast  of  new  developments 

•  NCO  identifies  authoritative  sources  of  information  on  Infantry  futures 
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Module 

Subject 

Behavioral  Outcomes 

Vehicle 

Maintenance 

•  NCO  follows  vehicle  maintenance  SOP  including  documentation  reqts 

•  NCO  maintains  or  provides  input  to  unit  maintenance  schedule 

•  NCO  routinely  monitors  maintenance  status  of  vehicles 

•  NCO  inspects  vehicles  as  part  of  equipment  readiness  program 

•  NCO  explains  vehicle  maintenance  schedule  &  procedures  to  Soldiers 

•  NCO  verifies  Soldiers  understand  their  vehicle  maintenance  duties 

•  NCO  coaches  Soldiers  in  proper  maintenance  procedures 

•  NCO  answers  Soldiers’  questions  in  accurate  and  timely  fashion 

•  NCO  supervises  execution  of  squad  level  maintenance  activities 

•  NCO  monitors  Soldiers’  maintenance  actions  and  provides  feedback 

•  NCO  verifies  implementation  of  proper  vehicle  maintenance  procedures 

•  NCO  identifies  maintenance  problems  and  takes  corrective  action 

•  NCO  mentors  new  NCOs  on  operator  vehicle  maintenance  duties 

•  NCO  stays  aware  of  changes  in  vehicle  maintenance  requirements 

•  NCO  integrates  new  systems  into  squad  vehicle  maintenance  program 

•  Soldiers  execute  proper  squad  level  maintenance  procedures 

Platoon  Sgt  Roles 
(Familiarization) 

•  NCO  explains  Platoon  Sergeant  duties  to  other  NCOs  and  Soldiers 

•  NCO  answers  Soldiers’  questions  in  accurate  and  timely  fashion 

•  NCO  identifies  authoritative  sources  of  information  on  Pit  Sgt  roles 

•  NCO  seeks  mentoring  from  Platoon  Sergeant 

•  NCO  assists  or  stands  in  for  Platoon  Sergeant  as  opportunities  arise 

WEAPONS 

Demolitions 

•  NCO  applies  demolition  fundamentals  in  training  exercises 

•  NCO  selects  appropriate  explosive  and  technique  for  situation 

•  NCO  executes  proper  steps  during  training  exercises 

•  NCO  adjusts  demolition  procedures  according  to  exercise  conditions 

•  NCO  gives  proper  commands/directions  to  Soldiers  at  training  site 

•  NCO  IDs  training  objectives  (Cdr,  intermediate,  and  opportunity  tasks) 

•  NCO  prepares  squad  training  plan  that  is  realistic  and  challenging 

•  NCO  develops  training  plan  IAW  TACSOP  and  changing  conditions 

•  NCO  develops  contingency  plan  for  weather,  time,  and  other  factors 

•  NCO  IDs  hazards/risks  and  implements  appropriate  control  measures 

•  NCO  requests  all  resources  (incl  explosives)  required  by  training  plan 

•  NCO  confirms  that  all  resources  are  available,  adjusts  if  necessary 

•  NCO  follows  range  occupation/setup  plan,  adjusts  as  necessary 

•  NCO  follows  training  execution  plan  and  adapts  to  changing  conditions 

•  NCO  manages  time  effectively  so  all  Soldiers  are  engaged  in  training 

•  NCO  implements  concurrent  training  plan  and  adapts  to  changes 

•  NCO  facilitates  learning  by  tolerating  failure  and  giving  timely  feedback 

•  NCO  implements  contingency  plan  immediately  upon  notification 

•  NCO  evaluates  performance  measures  IAW  established  task  standards 

•  NCO  conducts  AARs  that  cover  four  basic  elements 

•  NCO  uses  constructive  techniques  in  conducting  AARs 

•  NCO  follows  recovery  plan  and  adapts  to  changing  conditions 

•  Soldiers  execute  proper  demolition  procedures  during  training 

•  Squad/section  performs  to  standard  in  training  exercises 
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Module 

Subject 

Behavioral  Outcomes 

Small  Arms 
Proficiency* 

•  NCO  IDs  training  objectives  (Cdr,  intermediate,  and  opportunity  tasks) 

•  NCO  prepares  squad  training  plan  that  is  realistic  and  challenging 

•  NCO  develops  training  plan  IAW  checklist  and  changing  conditions 

•  NCO  follows  8-step  training  strategy 

•  NCO  develops  contingency  plan  for  weather,  time,  and  other  factors 

•  NCO  creates  lesson  outline  IAW  group  objectives 

•  NCO  chooses  correct  range  for  the  weapon  system 

•  NCO  IDs  hazards/risks  and  implements  appropriate  control  measures 

•  NCO  requests  all  resources  (incl  full  ammo)  required  by  training  plan 

•  NCO  confirms  that  all  resources  are  available,  adjusts  if  necessary 

•  NCO  follows  range  occupation/setup  plan,  adjusts  as  necessary 

•  NCO  follows  training  execution  plan  and  adapts  to  changing  conditions 

•  NCO  gives  proper  commands  to  students  on  firing  range 

•  NCO  coaches  Soldiers  in  proper  weapons  operation  and  safety 

•  NCO  coaches  Soldiers  in  marksmanship  fundamentals 

•  NCO  gives  signal  to  change  weapon  barrel  at  proper  time 

•  NCO  manages  time  effectively  so  all  Soldiers  are  engaged  in  training 

•  NCO  implements  concurrent  training  plan  and  adapts  to  changes 

•  NCO  facilitates  learning  by  tolerating  failure  and  giving  timely  feedback 

•  NCO  implements  contingency  plan  immediately  upon  notification 

•  NCO  evaluates  performance  measures  IAW  ARTEP  standards 

•  NCO  conducts  AARs  that  cover  four  basic  elements 

•  NCO  uses  constructive  techniques  in  conducting  AARs 

•  NCO  follows  recovery  plan  and  adapts  to  changing  conditions 

•  Soldiers  place  weapon  system  into  operation  correctly  and  fully 

•  Soldiers  apply  marksmanship  fundamentals  to  achieve  passing  score 

•  Soldiers  adjust  firing  procedures  according  to  environmental  conditions 

•  Soldiers  control  rate  of  fire  appropriately 

•  Soldiers  execute  proper  steps  to  clear  weapon 

COMBATIVES 

Basic  Techniques 

•  NCO  realistically  incorporates  combatives  into  training  plans 

•  NCO  confirms  that  all  equipment  and  safety  devices  are  available 

•  NCO  explains  combative  techniques  to  Soldiers  and  fellow  NCOs 

•  NCO  coaches  Soldiers  in  combative  techniques  and  skills 

•  NCO  shares  combative  experiences  with  Soldiers 

•  NCO  manages  time  effectively  so  all  Soldiers  are  engaged  in  training 

•  NCO  answers  Soldiers’  questions  in  accurate  and  timely  fashion 

•  NCO  identifies  authoritative  sources  of  information  on  combatives 

•  NCO  verifies  Soldiers  perform  proper  combative  techniques 

•  NCO  facilitates  learning  by  tolerating  failure  and  giving  timely  feedback 

•  NCO  evaluates  Soldier  performance  IAW  established  task  standards 

•  NCO  provides  feedback  on  combatives  performance  during  AARs 

•  Soldiers  execute  proper  combative  techniques  during  training  exercises 

•  Soldiers  perform  to  standard  in  practice  sessions 

•  Soldiers  adjust  combative  techniques  according  to  tactical  conditions 
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Module 

Subject 

Behavioral  Outcomes 

Basic  Fighting 
Strategy 

•  NCO  realistically  incorporates  combatives  into  training  plans 

•  NCO  confirms  that  all  equipment  and  safety  devices  are  available 

•  NCO  explains  combative  strategy  to  Soldiers  and  fellow  NCOs 

•  NCO  coaches  Soldiers  in  combative  fighting  strategy 

•  NCO  shares  fighting  experiences  with  Soldiers 

•  NCO  manages  time  effectively  so  all  Soldiers  are  engaged  in  training 

•  NCO  answers  Soldiers’  questions  in  accurate  and  timely  fashion 

•  NCO  identifies  authoritative  sources  of  information  on  combatives 

•  NCO  verifies  Armys  implement  proper  fighting  strategy 

•  NCO  facilitates  learning  by  tolerating  failure  and  giving  timely  feedback 

•  NCO  evaluates  Soldier  performance  IAW  established  task  standards 

•  NCO  provides  feedback  on  combatives  performance  during  AARs 

•  Soldiers  execute  proper  fighting  strategy  during  training  exercises 

•  Soldiers  perform  to  standard  in  practice  sessions 

•  Soldiers  adjust  fighting  strategy  according  to  tactical  conditions 

Basic  Drills 

•  See  Base  Techniques  and  Basic  Fighting  Strategy 

Advanced  Ground 
Fighting  Familiariz’n 

•  See  Basic  Techniques  and  Basic  Fighting  Strategy 

PLATOON 

TACTICS 

Platoon  Tactical 
Operations 

•  NCO  backbriefs  Pit  Ldr  to  convey  understanding  of  OPORD 

•  NCO  briefs  OPORD  to  squad  members  in  unit  training  exercises 

•  NCO  assists  Pit  Ldr/Sgt  in  executing  TLP  during  unit  training  exercises 

•  NCO  manages  time  to  meet  mission  milestones  in  unit  tng  exercises 

•  NCO  implements  TLP  to  achieve  mission  goals  in  unit  tng  exercises 

•  NCO  answers  Soldiers’  questions  in  accurate  and  timely  fashion 

•  NCO  identifies  authoritative  sources  of  information  on  orders  and  TLP 

•  NCO  works  to  improve  TLP  skills  based  on  AAR  feedback 

•  NCO  coaches  peers  re:  combat  orders,  TLP,  tactical  briefing,  etc. 

•  NCO  seeks  mentoring  from  Pit  Sgt  re:  combat  orders  and  TLP 

Tactical 

Questioning 

•  NCO  realistically  incorporates  tactical  questioning  into  training  plans 

•  NCO  explains  proper  procedures  to  Soldiers  and  fellow  NCOs 

•  NCO  demonstrates  use  of  proper  tactical  questioning  techniques 

•  NCO  coaches  Soldiers  in  tactical  questioning  procedures 

•  NCO  shares  operational  experiences  during  class  discussions 

•  NCO  answers  Soldiers’  questions  in  accurate  and  timely  fashion 

•  NCO  IDs  authoritative  sources  of  information  on  tactical  questioning 

•  NCO  verifies  Soldiers  follow  unit  TACSOP,  adjusts  training  as  needed 

•  NCO  facilitates  learning  by  tolerating  failure  and  giving  timely  feedback 

•  NCO  evaluates  performance  measures  IAW  TACSOP  and  training  plan 

•  NCO  provides  feedback  on  Soldier  performance  during  AARs 

•  Soldiers  perform  proper  tactical  questioning  during  training  exercises 

•  Soldiers  perform  to  standard  in  training  exercises 

•  Soldiers  adjust  procedures  according  to  tactical  conditions 
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Module 

Subject 

Behavioral  Outcomes 

Infantry  Battle 

Drills** 

•  NCO  IDs  training  objectives  (Cdr,  intermediate,  and  opportunity  tasks) 

•  NCO  prepares  squad  training  plan  that  is  realistic  and  challenging 

•  NCO  develops  training  plan  IAW  checklist  and  changing  conditions 

•  NCO  follows  8-step  training  strategy 

•  NCO  develops  contingency  plan  for  weather,  time,  and  other  factors 

•  NCO  creates  lesson  outline  IAW  group  objectives 

•  NCO  chooses  correct  training  site/range  for  the  training  objectives 

•  NCO  IDs  hazards/risks  and  implements  appropriate  control  measures 

•  NCO  requests  all  resources  (e.g.,  check  point  materials)  required 

•  NCO  confirms  that  all  resources  are  available,  adjusts  if  necessary 

•  NCO  follows  training  execution  plan  and  adapts  to  changing  conditions 

•  NCO  coaches  Soldiers  in  executing  battle  drills 

•  NCO  gives  proper  commands  to  squad  members  during  training 

•  NCO  executes  proper  leadership  techniques  during  training  exercises 

•  NCO  adjusts  procedures  according  to  tactical  conditions 

•  NCO  shares  operational  experiences  with  Soldiers 

•  NCO  manages  time  effectively  so  all  Soldiers  are  engaged  in  training 

•  NCO  implements  concurrent  training  plan  and  adapts  to  changes 

•  NCO  facilitates  learning  by  tolerating  failure  and  giving  timely  feedback 

•  NCO  implements  contingency  plan  immediately  upon  notification 

•  NCO  evaluates  performance  measures  IAW  ARTEP  standards 

•  NCO  conducts  AARs  that  cover  four  basic  elements 

•  NCO  uses  constructive  techniques  in  conducting  AARs 

•  NCO  follows  recovery  plan  and  adapts  to  changing  conditions 

•  Soldiers  perform  battle  drills  to  standard  in  training  exercises 

•  Squad/section  performs  battle  drills  to  standard  in  training  exercises 

•  Soldiers  adjust  procedures  according  to  tactical  conditions 

Tactical  Site 
Exploitation 

•  NCO  incorporates  tactical  site  exploitation  into  training  plans 

•  NCO  explains  proper  procedures  to  Soldiers  and  fellow  NCOs 

•  NCO  demonstrates  use  of  tactical  site  exploitation  techniques 

•  NCO  coaches  Soldiers  in  tactical  site  exploitation  procedures 

•  NCO  gives  proper  commands  to  squad  members  during  training 

•  NCO  executes  proper  leadership  techniques  during  training  exercises 

•  NCO  adjusts  procedures  according  to  tactical  conditions 

•  NCO  shares  operational  experiences  with  Soldiers 

•  NCO  answers  Soldiers’  questions  in  accurate  and  timely  fashion 

•  NCO  IDs  authoritative  sources  of  info  on  tactical  site  exploitation 

•  NCO  verifies  Soldiers  follow  unit  TACSOP,  adjusts  training  as  needed 

•  NCO  facilitates  learning  by  tolerating  failure  and  giving  timely  feedback 

•  NCO  evaluates  performance  measures  IAW  TACSOP  and  training  plan 

•  NCO  provides  feedback  on  Soldier  performance  during  AARs 

•  Soldiers  perform  tactical  site  exploitation  to  standard  during  training 

•  Squad/section  performs  to  standard  in  training  exercises 

•  Soldiers  adjust  procedures  according  to  tactical  conditions 
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Module 

Subject 

Behavioral  Outcomes 

Counterinsurgency 

(COIN) 

•  NCO  incorporates  COIN  tasks  and  procedures  into  training  plans 

•  NCO  explains  COIN  TTP  to  Soldiers  and  fellow  NCOs 

•  NCO  coaches  Soldiers  in  squad/section  level  COIN  procedures 

•  NCO  adjusts  procedures  according  to  tactical  conditions 

•  NCO  shares  operational  experiences  with  Soldiers 

•  NCO  answers  Soldiers’  questions  in  accurate  and  timely  fashion 

•  NCO  IDs  authoritative  sources  of  information  on  COIN  operations 

•  NCO  verifies  implementation  of  proper  COIN  procedures 

•  NCO  facilitates  learning  by  tolerating  failure  and  giving  timely  feedback 

•  NCO  evaluates  performance  measures  IAW  ARTEP  standards 

•  NCO  provides  feedback  on  Soldier  performance  during  AARs 

•  Soldiers  perform  COIN  procedures  to  standard  during  training 

•  Squad/section  performs  to  standard  in  training  exercises 

•  Soldiers  adjust  procedures  according  to  tactical  conditions 

Convoy  Operations 

•  NCO  incorporates  convoy  tasks  and  procedures  into  training  plans 

•  NCO  explains  convoy  TTP  to  Soldiers  and  fellow  NCOs 

•  NCO  coaches  Soldiers  in  squad/section  level  convoy  procedures 

•  NCO  adjusts  procedures  according  to  tactical  conditions 

•  NCO  shares  operational  experiences  with  Soldiers 

•  NCO  answers  Soldiers’  questions  in  accurate  and  timely  fashion 

•  NCO  IDs  authoritative  sources  of  information  on  convoy  operations 

•  NCO  verifies  implementation  of  proper  convoy  procedures 

•  NCO  facilitates  learning  by  tolerating  failure  and  giving  timely  feedback 

•  NCO  evaluates  performance  measures  IAW  ARTEP  standards 

•  NCO  provides  feedback  on  Soldier  performance  during  AARs 

•  Soldiers  perform  convoy  procedures  to  standard  during  training 

•  Squad/section  performs  to  standard  in  training  exercises 

•  Soldiers  adjust  procedures  according  to  tactical  conditions 

Counter  Sniper 
Operations 

•  NCO  incorporates  counter  sniper  tasks  and  procedures  into  training 
plans 

•  NCO  explains  counter  sniper  TTP  to  Soldiers  and  fellow  NCOs 

•  NCO  coaches  Soldiers  in  counter  sniper  TTP 

•  NCO  adjusts  procedures  according  to  tactical  conditions 

•  NCO  shares  operational  experiences  with  Soldiers 

•  NCO  answers  Soldiers’  questions  in  accurate  and  timely  fashion 

•  NCO  IDs  authoritative  sources  of  information  on  counter  sniper  TTP 

•  NCO  verifies  implementation  of  proper  counter  sniper  procedures 

•  NCO  facilitates  learning  by  tolerating  failure  and  giving  timely  feedback 

•  NCO  evaluates  performance  measures  IAW  Soldier  task  standards 

•  NCO  provides  feedback  on  Soldier  performance  during  AARs 

•  Soldiers  perform  counter-sniper  procedures  to  standard  during  training 

•  Squad/section  performs  to  standard  in  training  exercises 

•  Soldiers  adjust  procedures  according  to  tactical  conditions 
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Module 

Subject 

Behavioral  Outcomes 

Detainee  Operations 

•  NCO  incorporates  detainee  procedures  into  training  plans 

•  NCO  explains  detainee  operations  to  Soldiers  and  fellow  NCOs 

•  NCO  coaches  Soldiers  in  proper  detainee  processing  and  detention 

•  NCO  shares  operational  experiences  with  Soldiers 

•  NCO  answers  Soldiers’  questions  in  accurate  and  timely  fashion 

•  NCO  IDs  authoritative  sources  of  information  on  detainee  procedures 

•  NCO  verifies  implementation  of  proper  detainee  procedures 

•  NCO  facilitates  learning  by  tolerating  failure  and  giving  timely  feedback 

•  NCO  evaluates  performance  measures  IAW  Soldier  task  standards 

•  NCO  provides  feedback  on  Soldier  performance  during  AARs 

•  Soldiers  perform  detainee  procedures  to  standard  during  training 

•  Squad/section  performs  to  standard  in  training  exercises 

•  Soldiers  adjust  procedures  according  to  tactical  conditions 

*  Small  arms  proficiency  training  focuses  on  machine  guns  and  range  operations;  includes 
controlling  platoon  fires. 

**  Infantry  Battle  Drills  include  actions  on  contact,  road  block,  check  point  operations. 
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Behavioral  Taxonomy 


Category 

Representative  Behaviors 

Warfighting  Proficiency 

System  Proficiency 

•  Placing  weapon  into  operation  correctly  and  fully 

•  Executing  proper  steps  to  clear  weapon 

•  Initializing  and  operating  FBCB2  device  correctly 

•  Executing  proper  steps  during  training  exercises 

•  Performing  or  facilitating  system  troubleshooting 

•  Identifying  authoritative  sources  of  information  on  FBCB2 

Warrior  Proficiency 

•  Performing  land  navigation  to  Soldier  task  standards 

•  Performing  combative  techniques  to  standard  in  practice 

•  Achieving  acceptable  scores  on  weapons  qualification  tests 

•  Maintaining  battle  gear  under  diverse  weather  conditions 

Tactical  Proficiency 

•  Performing  forward  observer  procedures  correctly 

•  Implementing  TLP  to  achieve  mission  goals  in  exercises 

•  Identifying  authoritative  sources  of  TTP 

Collective  Proficiency 

•  Soldiers  executing  proper  land  navigation  procedures 

•  Soldiers  executing  proper  maintenance  procedures 

•  Soldiers  adjusting  procedures  according  to  tactical  conditions 

•  Squads/sections/crews  performing  to  standard  in  training 

•  Squads/sections/crews  achieving  proficiency  in  collective 
tasks  during  home  station  training 

•  Squads/sections/crews  achieving  proficiency  in  collective 
tasks  during  CTC  training 

Leadership 

Direct  Leadership 

•  Verifying  Soldiers  understand  their  duties/responsibilities 

•  Giving  proper  commands  to  Soldiers  while  training 

•  Supervising  execution  of  Soldiers’  maintenance  activities 

•  Verifying  application  of  proper  maintenance  procedures 

•  Briefing  OPORD  to  squad  members  in  training  exercises 

•  Managing  time  to  meet  mission  milestones  during  exercises 

Adaptive  Thinking 

•  Adjusting  task  execution  according  to  exercise  conditions 

•  Adjusting  training  plans  to  meet  changing  conditions 

•  Adjusting  training  execution  to  handle  weather  contingencies 

Knowledge  Sharing 

•  Sharing  FBCB2  knowledge/experiences  during  exercises 

•  Sharing  combative  experiences  during  practice  sessions 

•  Answering  Soldiers’  questions  about  Infantry  initiatives 
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Category 

Representative  Behaviors 

Custodianship 
(Personnel,  Equipment, 
Supplies) 

•  Implementing  end-of-training  recovery  plan  at  ranges 

•  Following  vehicle  maintenance  SOP,  incl  documentation 

•  Maintaining  or  providing  input  to  unit  maintenance  schedule 

•  Conducting  inspections  as  part  of  unit  readiness  program 

•  Taking  corrective  actions  to  solve  maintenance  problems 

•  Following  proper  procedures  for  property  accountability 

•  Ensuring  accuracy  of  Soldiers’  personnel  records 

Self-Development 

•  Describing  own  efforts  to  stay  abreast  of  Infantry  initiatives 

•  Staying  aware  of  changes  in  maintenance  requirements 

•  Working  to  improve  TLP  skills  based  on  AAR  feedback 

•  Seeking  mentoring  from  Platoon  Sergeant 

•  Assisting  or  standing  in  for  Platoon  Sergeant 

Training 

Teaching  and  Instructing 

•  Explaining  vehicle  maintenance  procedures  to  Soldiers 

•  Answering  Soldiers’  questions  accurately  and  promptly 

•  Facilitating  learning  by  tolerating  failure  of  squad  members 

•  Facilitating  learning  by  correcting  Soldiers  on  the  spot 

Coaching  and  Mentoring 

•  Coaching  Soldiers  in  land  navigation  procedures  and  skills 

•  Coaching  Soldiers  in  forward  observer  procedures 

•  Demonstrating  tactical  site  exploitation  procedures 

•  Demonstrating  tactical  questioning  techniques  to  Soldiers 

•  Mentoring  new  NCOs  on  operator  maintenance  duties 

Training  Management 
(Planning,  Preparing) 

•  Developing  weapon  training  plan  IAW  8-step  training  strategy 

•  Incorporating  tactical  questioning  into  training  plans 

•  Realistically  incorporating  new  equipment  into  tng  plans 

•  Identifying  hazards/risks  and  appropriate  control  measures 

•  Requesting  all  resources  required  to  implement  tng  plan 

•  Verifying  all  resources  are  in  place  prior  to  training 

Training  Operations 
(Executing,  Directing, 
Controlling) 

•  Implementing  training  execution  plan  on  time 

•  Confirming  proper  setup  of  FBCB2  and  Tactical  Internet 

•  Verifying  Soldiers  follow  FBCB2  procedures  of  TACSOP 

•  Managing  time  so  all  Soldiers  participate  in  training 

•  Implementing  control  measures  to  minimize  hazards/risks 

Training  Evaluation 
(Measuring  Performance, 
Providing  Feedback) 

•  Evaluating  performance  measures  IAW  ARTEP  T&EO 

•  Measuring  performance  IAW  Soldier  task  standards 

•  Conducting  AARs  that  cover  all  basic  elements 

•  Providing  feedback  on  FO  performance  during  AARs 

•  Including  digital  performance  aspects  in  AAR  discussions 
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NCO  Competence  Indicators 


Category 

Indicators 

Leadership 

Behavior 

•  Monitoring  the  morale,  discipline,  and  health  of  assigned  Soldiers 

•  Developing  subordinates  through  coaching,  mentoring,  and  counseling 

•  Receiving  respect  from  superiors,  peers,  and  subordinates 

•  Receiving  requests  for  advice  and  knowledge  from  superiors  &  peers 

•  Guiding  Soldiers  by  example  and  effective  role  modeling 

•  Fostering  teamwork  and  cohesion  among  subordinates  and  peers 

•  Displaying  strong  moral  character  and  Warrior  spirit 

Job 

Performance 

•  Holding  significant  duty  positions  of  progressively  greater  responsibility 

•  Performing  additional  duties  in  timely  and  effective  manner 

•  Communicating  orders  clearly,  correctly  (per  doctrine),  and  completely 

•  Assigning  tasks  to  subordinates  in  accordance  with  the  unit  mission 

•  Communicating  clear  guidance  to  subordinates  re:  task  performance 

•  Maintaining  proper  accountability  and  accurate  awareness  of  all 
personnel,  equipment,  and  supplies 

•  Taking  action  to  report  and  resolve  problems/deficiencies  with 
personnel,  equipment,  and  supplies 

•  Supervising  proper  maintenance  of  all  assigned  equipment 

•  Demonstrating  technical  and  tactical  competence  on  all  assigned 
combat  systems  (weapons,  vehicles,  FBCB2,  etc.)  and  TADSS 

•  Demonstrating  tactical  competence  in  executing  squad/section 
formations,  movements,  actions,  and  security 

•  Demonstrating  doctrinal  proficiency  on  squad  and  team  battle  drills 

•  Demonstrating  tactical  and  leadership  competence  during  operations 

•  Adjusting  or  adapting  TTP  to  align  with  METT-TC  and  TACSOP 

•  Planning,  preparing,  executing,  and  evaluating  the  training  of  Skill 

Level  1-2-3  tasks  (individual  and  Common) 

Unit  Readiness 

•  Maintaining  required  manning  and  qualification  levels  for  squad/team/ 
crew  (USR,  within  available  resources  and  unit  priorities) 

•  Correctly  reporting  acceptable  status  of  all  assigned  equipment  (USR 
R-level,  given  availability  of  parts  and  maintenance  support) 

•  Correctly  reporting  acceptable  training  status  of  Skill  Level  1-2-3  tasks 
(individual  and  Common) 

•  Achieving  acceptable  doctrinal  proficiency  (fully  trained  or  partially 
trained)  of  subordinate  elements  in  all  collective  tasks 

•  Achieving  acceptable  doctrinal  proficiency  of  subordinate  elements  in 
all  CTC  evaluations 

Primary  Sources:  (1 )  FM  3-21 .8,  The  Infantry  Rifle  Platoon  and  Squad ;  (2)  FM  7-22.7,  The 
Army  NCO  Guide ;  (3)  DA  Pamphlet  600-25,  US  Army  Noncommissioned  Officer  Professional 
Development  Guide ;  (4)  FM  6-22,  Army  Leadership — Competent,  Confident,  and  Agile ;  (5)  AR 
220-1 ,  Unit  Status  Reporting. 
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Final  Package  of  ALC  Impact  Metrics 


Data  Collection  Codes: 


Data  Source 

Collection  Technique  (final  letter  of  code) 

N  =  NCOs  (participants) 

R  =  Ratings  (self-anchored,  other-anchored) 

L  =  Leaders3 

W  =  Write-in  (on  questionnaire) 

P  =  Peers'3 

F  =  Focus  group  comments 

PS  =  Platoon  Sergeants 

UR  =  Unit  Records0 

3  Leaders  can  be  Pit  Sgts,  Pit  Ldrs,  Co  ISGs,  Co  Cdrs 
b  Peers  will  be  other  squad  leaders  within  NCO’s  company 

c  Unit  records  may  include  routine  testing  records,  training  records,  personnel  reports,  etc. 
Examples: 

•  NF  =  NCO  focus  group  comments 

•  PSW  =  platoon  sergeant  write-in  on  questionnaire 

•  PR  =  peer  ratings 


Measure 

Data  Elements  [Data  Collection  Codes] 

Sources 

Critical  and  Creative  Thinker 

NCO  adjusts  and 
executes  TTP  according 
to  tactical  conditions 
(METT-TC) 

•  Ratings  of  extent  of  ALC  practice  in  executing  TTP  adaptively 
[NR] 

•  Ratings  of  extent  to  which  ALC  learning  transfers  to  adjusting 
TTPorTSOP  [NR] 

•  Ratings  of  ALC  impact  on  NCO’s  adaptive  execution  of  TTP  or 
TSOP  [NR,  LR,  PR] 

•  Comments  re:  ALC  impact  on  NCO’s  adaptation  to  varying 
conditions  [NW,  NF,  LW,  LF,  PW,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  manages  time  to 
meet  mission  milestones 
during  operations 

•  Ratings  of  extent  to  which  ALC  events  provide  practice  on 
managing  time  [NR] 

•  Ratings  of  extent  to  which  ALC  learning  transfers  to  managing 
time  [NR] 

•  Ratings  of  ALC  impact  on  NCO’s  accomplishment  of  mission 
milestones  on  time  [NR,  LR,  PR] 

•  Comments  re:  ALC  impact  on  NCO’s  time  management  [NW,  NF, 
LW,  LF,  PW,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 
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Measure 

Data  Elements  [Data  Collection  Codes] 

Sources 

NCO  anticipates  and 
plans  for  the  unexpected 
by  thinking  ahead 

•  Ratings  of  extent  to  which  ALC  events  provide  practice  in  thinking 
and  planning  ahead  [NR] 

•  Ratings  of  extent  to  which  ALC  learning  transfers  to  planning  for 
the  unexpected  [NR] 

•  Ratings  of  ALC  impact  on  NCO’s  (1 )  planning  for  the  unexpected 
and  (2)  handling  unexpected  events  [NR,  LR,  PR] 

•  Comments  re:  ALC  impact  on  NCO’s  thinking  ahead  and  handling 
unexpected  events  [NW,  NF,  LW,  LF,  PW,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  formulates  lessons 
learned  based  on  own  and 
unit  experiences 

•  Ratings  of  extent  to  which  ALC  trains  NCOs  to  figure  out  lessons 
learned  [NR] 

•  Ratings  of  ALC’s  positive  impact  on  NCO’s  contributions  to  unit 
lessons  learned  [LR,  PR] 

•  Estimates  of  opportunities  to  derive  lessons  learned  [LW,  LF] 

•  Examples  of  lessons  learned  [NW,  NF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  realistically  incorp¬ 
orates  new  equipment  and 
TTP  into  unit  operations 

•  Ratings  of  ALC’s  positive  impact  on  NCO’s  ability  to  incorporate 
new  equipment  and  TTP  into  unit  operations  [LR,  PR] 

•  List  of  new  equipment  and  TTP  received  [LW,  LF] 

•  Examples  of  accomplishments  incorporating  new  equipment/TTP 
into  unit  operations  [NW,  NF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

Warrior  Leader  -  Military  Leadership 

NCO  fosters  teamwork 
and  positive  climate  within 
the  squad 

•  Ratings  of  squad’s  improvement  in  (1)  teamwork,  (2)  morale,  (3) 
cohesion,  and  (4)  unit  climate  due  to  skills  developed  in  ALC  [NR, 
LR,  PR] 

•  Ratings  of  squad’s  decrease  in  (1)  absenteeism,  (2)  turnover,  and 
(3)  disciplinary  actions  due  to  skills  from  ALC  [NR,  LR] 

•  Examples  (positive  &  negative)  re:  teamwork  and  climate  within 
the  squad  [NW,  NF] 

•  Adverse  personnel  incidents  (AWOL,  UCMJ  actions,  etc.)  [UR] 

•  Siebold  &  Kelly’s  (1988)  pit  cohesion  survey  (selected  items?) 

•  ARI’s  Command  Climate  Survey  for  TO&E  units  (selected  items?) 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

•  Unit  Records 

Subordinates  understand 
their  everyday  duties  and 
responsibilities 

•  Ratings  of  improvement  in  subordinates’  job  understanding  due  to 
NCO  skills  learned  in  ALC  [NR] 

•  Ratings  of  improvement  needed  in  subordinates’  job 
understanding  [NR] 

•  Examples  (positive  &  negative)  of  subordinates’  job 
understanding  [NW,  NF] 

.  NCOs 

NCO  communicates  clear 
guidance  to  subordinates 
re:  task  performance 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  ability  to  give 
subordinates  clear  guidance  re:  task  performance  [NR,  LR,  PR] 

•  Ratings  of  improvement  needed  re:  NCO  guidance  on  task 
performance  provided  to  subordinates  [NR] 

•  Examples  (positive  &  negative)  of  NCO  guidance  to  subordinates 
[NW,  NF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  conveys  mission 
orders  to  Soldiers  clearly, 
correctly,  and  completely 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  ability  to  convey 
orders  to  subordinates  [NR,  LR,  PR] 

•  Ratings  of  improvement  needed  in  NCO’s  process  of  conveying 
orders  [NR] 

•  Examples  (positive  &  negative)  of  clear/unclear  orders  [NF,  LF, 

PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 
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Measure 

Data  Elements  [Data  Collection  Codes] 

Sources 

NCO  ensures  subordi¬ 
nates  treat  others  with 
respect,  including  local 
citizens 

•  Ratings  of  ALC-attributed  improvement  in  respect  shown  by 

NCO’s  subordinates  [NR,  LR,  PR] 

•  Ratings  of  subordinates’  respect  for  (1 )  each  other,  (2)  Family 
members,  and  (3)  community  members  [NR] 

•  Examples  (positive  &  negative)  of  respect  shown  by  squad 
members  [NF] 

•  Adverse  incidents  related  to  lack  of  respect  (UCMJ  actions,  etc.) 

.  NCOS 

•  Unit  Leaders 

•  Unit  Peers 

•  Unit  Records 

Warrior  Leader  -  Warrior  Competencies 

NCO  applies  proper  TTP 
in  executing  collective 
tasks  and  battle  drills 

•  Ratings  of  extent  to  which  ALC  events  provide  practice  on 
implementing  TTP  [NR] 

•  Ratings  of  improvement  in  NCO’s  knowledge  of  doctrine  and  TTP 
resulting  from  ALC  learning  [NR,  LR,  PR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  application  of 
TTP/TSOP  during  (1)  battle  drills  and  (2)  collective  tasks  [NR,  LR, 
PR] 

•  Comments  re:  ALC  impact  on  NCO’s  implementation  of  TTP  [NW, 
NF,  LW,  LF,  PW,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  properly  employs  all 
assigned  and  available 
equipment  (weapons, 
radios,  computers, 
vehicles,  etc.) 

•  Ratings  of  extent  to  which  ALC  events  provide  practice  in 
employing  equipment  [NR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  ability  to  select 
proper  weapon(s)  for  a  given  mission  [NR,  LR,  PR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  ability  to  employ 
assigned  (1)  weapons,  (2)  radios,  (3)  vehicles,  (4)  battle 
command  systems,  (5)  special  mission  equipment,  and  (6) 
individual  equipment  [NR,  LR,  PR] 

•  Comments  re:  ALC  impact  on  NCO’s  (1)  knowledge  and  (2) 
employment  of  assigned  and  available  equipment  [NW,  NF,  LW, 

LF,  PW,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  achieves  acceptable 
level  of  proficiency  for 
individual  tasks/skills  (e.g., 
APFT,  weapons  qualifica¬ 
tion,  radio  and  computer 
operations) 

•  Ratings  of  extent  to  which  ALC  provides  practice  on  (1 )  physical 
fitness,  (2)  weapons  qualification,  (3)  radio  operations,  (4)  other 
Warrior  Tasks,  and  (5)  ABCS  operations  [NR] 

•  Ratings  of  extent  to  which  ALC  improved  NCO’s  Warrior 
competencies  [NR,  LR,  PR] 

•  Comments  re:  ALC  impact  on  NCO’s  level  of  Warrior  Task 
proficiency  [NW,  NF,  LW,  LF,  PW,  PF] 

•  NCO  scores  (post-ALC)  on  APFT  and  weapons  qualification 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

•  Unit  Records 

Leader  Developer  -  Counseling,  Coaching  and  Mentoring 

NCO  answers  Soldiers’ 
questions  and  shares 
knowledge  and  experi¬ 
ences 

•  Ratings  of  ALC-attributed  improvement  in  ability  to  answer 

Soldiers’  questions  [NR] 

•  Ratings  of  ALC  impact  on  NCO’s  sharing  of  knowledge  and 
experiences  [NR,  LR,  PR] 

•  Ratings  of  quality  and  quantity  of  information,  knowledge  and 
answers  provided  to  subordinates  by  NCO  [NR] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  guides  and  develops 
subordinates  by  coaching, 
mentoring,  counseling  and 
role  modeling 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  coaching/ 
mentoring/counseling/role  modeling  [NR,  LR,  PR] 

•  Ratings  of  subordinates’  positive  feedback  re:  coaching/ 
mentoring/counseling  provided  by  NCO  [NR] 

•  Positive  examples  of  NCO’s  coaching/mentoring/counseling/role 
modeling  [NF,  LF,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

E-3 


Measure 

Data  Elements  [Data  Collection  Codes] 

Sources 

NCO  reinforces  ethical 
standards  of  behavior 
among  subordinates 

•  Ratings  of  extent  to  which  ALC  provides  training  on  ethical 
standards  [NR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  ability  to 
promote  ethical  behavior  among  subordinates  [NR,  LR] 

•  Ratings  of  ALC’s  positive  impact  on  ethical  behavior  within  NCO’s 
squad  [LR,  PR] 

•  Ratings  of  subordinates’  positive  feedback  re:  ethical  behavior 
within  squad  [NR] 

.  NCOS 

•  Unit  Leaders 

•  Unit  Peers 

Leader  Developer  -  Training  Subordinates 

NCO  develops,  prepares, 
and  executes  realistic 
training  plans  for  Skill 

Level  1-2-3  tasks  (individ¬ 
ual  and  Common) 

•  Ratings  of  extent  of  ALC  practice  in  (1 )  developing,  (2)  preparing, 
and  (3)  executing  realistic  squad  training  [NR] 

•  Ratings  of  ALC-related  improvement  in  NCO’s  ability  to  develop/ 
prepare/execute  realistic  squad  training  [NR,  LR,  PR] 

•  Comments  re:  ALC  impact  on  NCO’s  development/preparation/ 
execution  of  squad  training  [NW,  NF,  LW,  LF,  PW,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  manages  training 
activities  to  optimize 
participation  and  safety 

•  NCO  ratings  of  extent  to  which  ALC  provides  practice  in 
managing  training  activities  [NR] 

•  Ratings  of  ALC-attributed  improvement  in  squad  training  activities 
re:  (1)  effectiveness,  (2)  Soldier  participation,  and  (3)  safety  [NR, 
LR,  PR] 

•  Ratings  of  subordinates’  positive  feedback  re:  successful  and 
effective  training  [NR] 

•  Comments  re:  ALC  impact  on  NCO’s  management  of  squad 
training  activities  [NW,  NF,  LW,  LF,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  explains/demon¬ 
strates  Soldier  tasks 

•  Ratings  of  extent  to  which  ALC  provides  practice  in  explaining 
and  demonstrating  Soldier  tasks  [NR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  ability  to 
explain/demonstrate  Soldier  tasks  [NR,  LR,  PR] 

•  Ratings  of  subordinates’  positive  feedback  re:  NCO’s  explanation 
and  demonstration  of  Soldier  tasks  [NR] 

•  Comments  re:  ALC  impact  on  NCO’s  explanation  and  demon¬ 
stration  of  Soldier  tasks  [NW,  NF,  LW,  LF,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  properly  evaluates 
performance  of  subordi¬ 
nate  Soldiers  on  individual 
and  Common  Tasks,  and 
provides  constructive 
feedback 

•  Ratings  of  extent  to  which  ALC  provides  practice  in  (1 )  evaluating 
Soldier  performance  and  (2)  providing  feedback  [NR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  ability  to  (1) 
properly  evaluate  Soldier  performance  and  (2)  provide 
constructive  feedback  [NR,  LR,  PR] 

•  Comments  re:  ALC  impact  on  NCO’s  ability  to  evaluate  Soldier 
performance  and  provide  feedback  [NW,  NF,  LF,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  utilizes  TADSS 
appropriately  for  training 

•  Ratings  of  extent  to  which  ALC  provides  opportunities  to  employ 
relevant  TADSS  [NR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  ability  to  utilize 
TADSS  appropriately  and  effectively  [NR,  LR,  PR] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

Leader  Developer  -  Shaping  Unit  Performance 

Soldiers  perform  individual 
and  Common  Tasks  to 
standard 

•  Ratings  of  squad’s  improvement  in  individual  and  Common  Task 
performance  due  to  skills  NCO  learned  in  ALC  [NR,  LR,  PR] 

•  Ratings  of  positive  feedback  from  superiors  and  peers  re:  ALC- 
attributed  improvement  of  squad’s  task  performance  [NR] 

•  Comments  re:  ALC  impact  on  squad’s  performance  of  individual 
and  Common  Tasks  [NW,  NF,  LW,  LF,  PF] 

•  Subordinates’  scores  on  individual  and  Common  Task  tests  [UR] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

•  Unit  Records 
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Measure 

Data  Elements  [Data  Collection  Codes] 

Sources 

Squad/section/team 
executes  collective  tasks 
and  battle  drills  to 
standard 

•  Ratings  of  squad’s  improvement  in  collective  task  and  battle  drill 
performance  due  to  skills  NCO  learned  in  ALC  [NR,  LR,  PR] 

•  NCO  ratings  of  positive  feedback  from  superiors  and  peers  re: 
ALC-attributed  improvement  of  squad’s  collective  task  and  battle 
drill  performance  [NR] 

•  Comments  re:  ALC  impact  on  squad’s  performance  of  collective 
tasks  and  battle  drills  [NW,  NF,  LW,  LF,  PF] 

•  Subordinates’  scores  on  collective  tasks  and  Battle  Drills  [UR] 

.  NCOS 

•  Unit  Leaders 

•  Unit  Peers 

•  Unit  Records 

Soldiers  achieve  accept¬ 
able  APFT  and  weapons 
qualification  scores 

•  Ratings  of  squad’s  improvement  in  (1 )  APFT  and  (2)  weapons 
qualification  performance  due  to  skills  NCO  learned  in  ALC  [NR, 

LR,  PR] 

•  NCO  ratings  of  positive  feedback  from  superiors  and  peers  re: 
ALC-attributed  improvement  of  Soldiers’  performance  in  (1)  APFT 
and  (2)  weapons  qualification  events  [NR] 

•  Comments  re:  ALC  impact  on  squad’s  performance  in  APFT  and 
weapons  qualification  events  [NW,  NF,  LW,  LF,  PF] 

•  Subordinates’  scores  on  APFT  and  weapons  qualification  [UR] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

•  Unit  Records 

Soldiers  properly  employ 
assigned  equipment 
(weapons,  radios, 
computers,  vehicles,  etc.) 

•  Ratings  of  ALC-attributed  improvement  in  squad’s  employment  of 
assigned  (1)  weapons,  (2)  radios,  (3)  vehicles,  (4)  battle  com¬ 
mand  systems,  (5)  special  mission  equipment,  and  (6)  individual 
equipment  [NR,  LR,  PR] 

•  Comments  re:  ALC  impact  on  squad’s  (1 )  knowledge  and  (2) 
employment  of  assigned  equipment  [NF,  LF,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

Leader  Developer  -  Expanding  Own  Competencies 

NCO  works  to  improve 
technical,  tactical,  and 
leadership  skills  based  on 
supervisor  guidance/ 
counseling  and  AAR 
feedback 

•  Ratings  of  self-development  skills  learned  in  ALC  [NR] 

•  Ratings  of  ALC-influenced  ability  to  improve  professional  skills  - 

(1)  tactical,  (2)  leadership,  and  (3)  technical  [NR] 

•  Ratings  of  ALC  impact  on  NCO’s  professional  development 
activities  [NR,  LR,  PR] 

•  Ratings  of  positive  feedback  from  superiors  and  peers  re:  ALC- 
attributed  improvement  in  NCO’s  professional  skills  -  (1)  tactical, 

(2)  leadership,  and  (3)  technical  [NR] 

•  Listing  of  professional  development  activities  since  ALC  [NW] 

•  Comments  re:  ALC  impact  on  NCO’s  professional  development 
goals  and  activities  [NW,  NF,  LW,  LF,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  seeks  mentoring 
from  Platoon  Sergeant 

•  Ratings  of  ALC-influenced  decision(s)  to  seek  Platoon  Sergeant’s 
mentoring  [NR] 

•  Listing  of  mentoring  activities  [NW,  PSW] 

•  Ratings  of  ALC  impact  on  NCO’s  openness  to  mentoring  [PSR] 

•  Comments  re:  ALC  impact  on  NCO’s  efforts  to  seek  mentoring 
[NW,  NF,  PSW,  PSF] 

.  NCOs 
•  Pit  Sgts 

NCO  leads  and/or  partici¬ 
pates  in  professional 
development  sessions 

•  Ratings  of  ALC-influenced  decision(s)  to  lead  or  participate  in  unit 
professional  development  sessions  [NR] 

•  Ratings  of  ALC  impact  on  NCO’s  efforts  to  lead  or  participate  in 
unit  professional  development  sessions  [NR,  PSR,  LR,  PR] 

•  Listing  of  unit  professional  development  sessions  in  which  NCO 
led  or  participated  [NW,  NF,  PSF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 
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Measure 

Data  Elements  [Data  Collection  Codes] 

Sources 

NCO  assists  or  stands  in 
for  Platoon  Sergeant 

•  Ratings  of  ALC-influenced  decision(s)  to  learn  by  (1 )  assisting  or 
(2)  standing  in  for  Pit  Sgt  [NR] 

•  Ratings  of  ALC  impact  on  NCO’s  efforts  to  assist  or  stand  in  for 
Platoon  Sergeant  [NR,  PSR] 

•  Listing  of  NCO’s  efforts  to  assist  or  stand  in  for  Pit  Sgt  [NW,  PSW] 

.  NCOS 
•  Pit  Sgts 

Ambassador  -  All  Environments 

Soldiers  show  respect  for 
social/cultural  standards 
of  the  local  community 

•  Ratings  of  extent  to  which  ALC  provides  training  on  cultural/ 
community  respect  [NR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  ability  to  promote 
cultural/community  respect  among  subordinates  [NR] 

•  Ratings  of  squad’s  improvement  in  respect  shown  toward  local 
community  due  to  skills  NCO  learned  in  ALC  [NR,  LR,  PR] 

•  NCO  ratings  of  positive  feedback  from  subordinates,  superiors  and 
peers  re:  ALC-attributed  improvement  of  squad’s  social/  cultural 
respect  [NR] 

•  Comments  re:  ALC  impact  on  squad’s  respect  shown  toward  local 
community  [NF] 

•  Examples  (positive  &  negative)  of  squad’s  respect  shown  toward 
local  community  [NF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  ensures 
subordinates  represent 
Army  well  among  non- 
Army  elements  and  local 
community 

•  Ratings  of  subordinates’  improvement  as  Army  representatives 
among  local  community  due  to  NCO’s  learning  in  ALC  [NR,  LR, 

PR] 

•  Ratings  of  positive  feedback  from  subordinates,  superiors  and 
peers  re:  ALC-attributed  improvement  of  squad’s  positive  impact 
on  local  community  [NR] 

•  Comments  re:  ALC  influence  on  squad’s  positive  impact  on  local 
community  [NF,  LF,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  establishes  and 
maintains  contact  with 
non-Army  colleagues  and 
community  leaders 

•  Ratings  of  relevance  of  ALC  training  to  networking  with  (1) 
colleagues  in  other  services  and  (2)  community  leaders  [NR] 

•  Ratings  of  ALC-influenced  decisions/efforts  to  network  with  (1) 
colleagues  in  other  services  and  (2)  community  leaders  (NR,  LR, 
PR] 

•  Ratings  of  positive  feedback  from  superiors  and  peers  re:  ALC- 
attributed  improvement  in  NCO’s  interactions  with  (1)  colleagues 
in  other  services  and  (2)  community  leaders  [NR] 

•  Listing  of  NCO  contacts  with  (1 )  colleagues  in  other  services  and 
(2)  community  leaders  [NW] 

•  Comments  re:  ALC  influence  on  NCO’s  external  networking  [NF, 

LF,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  leads  and/or 
participates  in  joint, 
unified,  and  community 
engagement  activities 

•  Ratings  of  extent  to  which  ALC  training/events  involve  joint, 
unified,  and  community  aspects  [NR] 

•  Ratings  of  ALC-influenced  decisions/efforts  by  NCO  to  lead  or 
participate  in  multi-agency  activities  -  (1 )  joint,  (2)  unified,  and  (3) 
local  community  [NR,  PSR,  LR,  PR] 

•  Listing  of  activities/events  in  which  NCO  led  or  participated  -  (1) 
joint,  (2)  unified,  and  (3)  local  community  [NW,  PSF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

•  Pit  Sgts 
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Measure 

Data  Elements  [Data  Collection  Codes] 

Sources 

Ambassador  -  Operational  Environment 

NCO  develops  and  applies 
cultural  awareness  in 
combat  operations 

•  Ratings  of  extent  to  which  ALC  training/events  enhance  cultural 
awareness  [NR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  cultural 
awareness  and  understanding  [NR,  LR,  PR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  application  of 
cultural  knowledge  in  combat  operations  [NR,  LR,  PR] 

•  Comments  re:  ALC  impact  on  NCO’s  understanding  and 
application  of  cultural  awareness  [NF,  LF,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  incorporates 
community  impact  into 
planning  and  leading 
combat  operations 

•  Ratings  of  extent  to  which  ALC  provides  practice  in  considering 
community  impact  while  planning/leading  combat  ops  [NR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  under-standing 
and  accounting  for  community  factors  in  planning/  executing 
combat  operations  [NR,  LR,  PR] 

•  Comments  re:  ALC  influence  on  NCO’s  consideration  of 
community  factors  in  combat  operations  [NF,  LF,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

Resource  Manager 

NCO  maintains  aware¬ 
ness  of  and  proper 
accountability  for  all 
personnel,  equipment, 
and  supplies 

•  Ratings  of  extent  to  which  ALC  trains  NCOs  to  manage  unit 
resources  [NR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  control  and 
mgmt  of  (1 )  personnel,  (2)  equipment,  (3)  supplies  [NR,  LR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  accounting  and 
reporting  re:  (1)  personnel,  (2)  equipment,  and  (3)  supplies  [NR, 

LR] 

•  Comments  re:  ALC  influence  on  NCO’s  management  of 
resources  assigned  to  the  squad  [NW,  NF,  LW,  LF] 

.  NCOs 

•  Unit  Leaders 

NCO  takes  action  to 
report  and  resolve 
problems/deficiencies  with 
personnel,  equipment, 
and  supplies 

•  Ratings  of  extent  to  which  ALC  training  events  involve  practice  on 
solving  resource  problems  [NR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  identification 
and  solution  of  problems  with  (1 )  personnel,  (2)  equipment,  and 
(3)  supplies  [NR,  LR] 

•  Ratings  of  positive  feedback  from  superiors  re:  ALC-attributed 
improvement  in  status  of  (1)  personnel,  (2)  equipment,  and  (3) 
supplies  in  NCO’s  squad  [NR] 

•  Comments  re:  ALC  influence  on  NCO’s  performance  in  resolving 
problems/deficiencies  with  resources  [NF,  LF] 

.  NCOs 

•  Unit  Leaders 

NCO  ensures  all  assigned 
equipment  is  maintained 
properly 

•  Ratings  of  extent  to  which  ALC  training  events  involve  practice  on 
managing  equipment  maintenance  [NR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  management  of 
equipment  maintenance  activities  [NR,  LR] 

•  Ratings  of  positive  feedback  from  superiors  re:  ALC-attributed 
improvement  of  equipment  maintenance  in  NCO’s  squad  [NR] 

•  Comments  re:  ALC  impact  on  NCO’s  management  of  equipment 
maintenance  activities  [NW,  NF,  LW,  LF] 

.  NCOs 

•  Unit  Leaders 

Soldiers  execute  and 
document  proper 
maintenance  procedures 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  supervision  of 
subordinates’  execution  and  documentation  of  maintenance 
duties  [NR,  LR] 

•  Ratings  of  ALC-attributed  improvement  in  documentation  of 
equipment  maintenance  in  NCO’s  squad  [LR] 

•  Ratings  of  positive  feedback  from  superiors  re:  ALC-attributed 
improvement  in  documentation  of  equipment  maintenance  in 

NCO’s  squad  [NR] 

.  NCOs 

•  Unit  Leaders 
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Measure 

Data  Elements  [Data  Collection  Codes] 

Sources 

Global  Impact 

NCO’s  overall  abilities  and 
performance  improve  as  a 
result  of  ALC 

•  Ratings  of  overall  relevance  of  ALC  training  to  NCO’s  job 
performance  in  the  unit  [NR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  overall  (1) 
tactical  skills,  (2)  leadership  skills,  (3)  technical  skills,  (4)  people 
skills,  (5)  attitudes,  and  (5)  motivation  [NR,  LR,  PR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  overall  (1) 
behavior,  (2)  effectiveness,  (3)  productivity,  and  (4)  efficiency 
[NR,  LR,  PR] 

•  Ratings  of  ALC-attributed  improvement  in  NCO’s  (1)  work  habits, 

(2)  performance,  and  (3)  unit  contributions  [NR,  LR,  PR] 

•  Ratings  of  ALC-attributed  increase  in  (1)  appreciation  of  NCO 
importance,  (2)  commitment  to  Army  goals,  and  (3)  personal 
identification  with  the  Army  [NR] 

•  Ratings  of  ALC-related  improvement  in  NCO’s  (1)  performance 
as  a  squad  leader,  (2)  abilities  as  a  trainer,  and  (3)  task 
performance  [NR,  PSR] 

•  Ratings  of  extent  to  which  skills  learned  in  ALC  transfer  to  job 
performance  (1)  at  home  station  and  (2)  while  deployed  [NR] 

•  Ratings  of  positive  feedback  from  superiors,  peers,  and  subordi¬ 
nates  re:  ALC  impact  on  own  (1)  competencies,  (2)  work  habits, 

(3)  performance,  and  (4)  contributions  to  the  unit  [NR] 

•  Comments  re:  ALC  impact  on  NCO’s  competencies,  work  habits, 
commitment,  motivation,  and  performance  {NW,  LW,  PW] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO’s  contributions  to  the 
unit  increase  as  a  result  of 
ALC  training 

•  Ratings  of  ALC’s  positive  impact  on  NCO’s  standing  in  the  eyes 
of  (1)  unit  leaders,  (2)  unit  peers,  and  (3)  subordinates  [NR,  LR, 

PR] 

•  Ratings  of  positive  feedback  from  superiors,  peers,  and 
subordinates  re:  ALC  benefits  to  NCO  [NR] 

•  Ratings  of  ALC’s  positive  impact  on  NCO’s  contributions  to  the 
unit  [NR,  LR,  PR] 

•  Comments  on  ALC  benefits  to  NCO  [NW,  NF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

NCO  uses  ALC  learning  to 
improve  conditions  in  the 
unit 

•  Ratings  of  extent  to  which  ALC  learning  enables  NCO  to  improve 
unit  training  and  operations  [NR,  LR,  PR] 

•  Ratings  of  extent  to  which  “soft  skills”  learned  in  ALC  (such  as 
people  skills)  help  NCO  to  improve  unit  training  and  ops  [NR] 

•  Ratings  of  extent  to  which  the  unit  environment  (1)  allows  and  (2) 
encourages  NCO  to  apply  knowledge  and  skills  learned  in  ALC 
[NR] 

•  Ratings  of  extent  to  which  NCO  is  recognized  for  using  skills 
learned  in  ALC  within  the  unit  [NR,  PSR] 

•  Ratings  of  extent  to  which  unit  leaders  view  ALC  training  as  a 
positive  factor  in  unit  operations  [NR] 

•  Ratings  of  extent  to  which  unit  leaders  view  time  NCOs  spend  in 
ALC  as  time  well  spent  [NR] 

•  Ratings  of  extent  to  which  ALC  enables  NCO  to  become  a  more 
effective  agent  of  change  within  the  unit  [NR,  LR,  PR] 

•  Ratings  of  extent  to  which  NCO  had  to  overcome  obstacles  to 
apply  ALC  learning  within  the  unit  [NR,  PR] 

•  Comments  on  NCO  efforts  to  improve  unit  training  and 
operations  by  applying  ALC  learning  [NF,  LF,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 
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Measure 

Data  Elements  [Data  Collection  Codes] 

Sources 

Squad’s  overall  perform¬ 
ance  improves  due  to 

NCO’s  learning  in  ALC 

•  Ratings  of  ALC’s  positive  impact  on  squad’s  (1)  drive  for 
excellence,  (2)  productivity,  and  (3)  effectiveness  [NR,  PSR,  LR] 

•  Ratings  of  ALC’s  positive  impact  on  squad’s  standing  in  the  eyes 
of  (1 )  unit  leaders,  (2)  unit  peers,  and  (3)  subordinates  [NR] 

•  Ratings  of  ALC’s  positive  impact  on  squad’s  (1)  overall  perform¬ 
ance  and  (2)  contributions  to  the  unit  [NR,  LR,  PR] 

•  Ratings  of  positive  feedback  from  superiors,  peers,  and  subordi¬ 
nates  re:  ALC  benefits  to  squad  [NR] 

•  Squad  and  platoon  performance  ratings  by  CTC  O/Cs  [UR] 

•  Comments  on  ALC  benefits  to  squad  [NW,  NF,  LW,  LF,  PF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 

•  CTC  Take- 
Home  Pkgs 

Unit  conditions  influence 
NCOs’  application  of  ALC 
learning 

•  Ratings  of  time  available  to  (1)  utilize  knowledge/skills  learned  in 
ALC  and  (2)  change  the  way  things  are  done  [NR,  PR] 

•  Ratings  of  opportunities  to  (1)  utilize  knowledge/skills  learned  in 
ALC  and  (2)  change  the  way  things  are  done  [NR,  PR] 

•  Ratings  of  extent  to  which  other  requirements  take  priority  over 
applying  ALC  knowledge  and  skills  [NR,  PR] 

•  Ratings  of  extent  to  which  unit  workload  allows  NCO  to  use 
knowledge  and  skills  learned  in  ALC  [NR,  PR] 

•  Ratings  of  extent  to  which  certain  ALC-acquired  knowledge  and 
skills  don’t  apply  to  (1 )  NCO’s  job  performance  and  (2) 
deployment  requirements  [NR,  LR,  PR] 

•  Ratings  of  extent  to  which  NCO  is  allowed  to  use  ALC-acquired 
knowledge/skills  to  change  unit  training  and  operations  [NR,  PR] 

•  Ratings  of  extent  to  which  NCO  is  encouraged  to  use  ALC- 
acquired  learning  to  change  the  way  things  are  done  [NR,  PR] 

•  Ratings  of  extent  to  which  NCO  is  sure  it  is  OK  to  (1 )  use  certain 
ALC-acquired  knowledge/skills  or  (2)  change  unit  procedures 
[NR,  PR] 

•  Ratings  of  extent  to  which  NCO  has  been  told  not  to  (1 )  use 
certain  ALC-acquired  knowledge/skills  or  (2)  change  unit 
procedures  [NR,  PR] 

•  Ratings  of  extent  to  which  NCO’s  supervisor  thinks  it  is  effective 
to  use  ALC-acquired  knowledge  and  skills  [NR,  PR] 

•  Ratings  of  extent  to  which  certain  techniques  or  procedures 
learned  in  ALC  are  different  from  the  unit’s  SOP  [NR,  PR] 

•  Comments  on  ALC  benefits  to  NCO  [NW,  NF] 

.  NCOs 

•  Unit  Leaders 

•  Unit  Peers 
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PURPOSE 


This  document: 

•  Provides  the  blueprint  for  ARI’s  evaluation  of  11BALC  impact 

•  Lays  out  comprehensive  evaluation  goals  and  objectives 

•  Outlines  the  design,  technical  approach,  and  sampling  strategy 

•  Establishes  a  systematic  measurement  strategy  and  framework 

•  Considers  parameters  for  building  a  data  collection  timeline 
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BACKGROUND 

•  The  POI  for  Infantry  ALC  Phase  II  (resident)  is  changing 

•  The  POI  has  new  focus,  revamped  contents,  shorter  duration 

•  The  transitional  POI  is  expected  to  lock-in  by  June  2009 

•  The  schoolhouse  needs  effectiveness  data  for  program  decisions 

•  The  evaluation  will  focus  on  ALC  impact  in  operational  units 

•  Early  work  identified  key  behaviors  and  developed  impact  metrics 

•  Evaluation  planning  addressed  design,  sampling,  methods,  tools,  etc. 

•  Impact  metrics  are  populating  data  collection  instruments 

•  Data  collection,  analysis  and  documentation  are  slated  for  FY10 
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EVALUATION  GOALS 


•  Purpose: 

•  Inform  schoolhouse  on  effectiveness  of  new  11 B  ALC  Phase  II 

•  Support  decisions  re:  revision  and  implementation  of  new  POI 

•  Goals: 

•  Provide  course  architects  and  managers  a  view  of  ALC’s  practical  impact 

•  Give  FORSCOM  units  a  means  of  communicating,  sharing,  and  influencing 

•  Illuminate  issues  re:  contents  and  implementation  of  11 B  ALC  POI 

•  Capture  unit  stakeholders’  suggestions  for  improving  the  POI 

•  Arm  curriculum  managers  with  empirical  data  to  justify  program  decisions 

•  Validate  11 B  ALC  revision  as  part  of  NCOES  modernization  strategy 

•  Promote  utilization  of  11 B  ALC  learning  among  operational  units 

•  Raise  awareness  of  ALC  benefits  and  payoff  among  FORSCOM  leaders 

•  Pave  the  way  for  continuing  program  evaluation 
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EVALUATION  OBJECTIVES 

•  Assess  frequency  and  relevance  of  Phase  II  training  opportunities 

•  Measure  improvement  in  proficiency  on  Infantry  NCO  competencies 

•  Quantify  NCO  learning  re:  knowledge,  skills,  and  attributes  (KSAs) 

•  Examine  NCO  perceptions  of  11 B  ALC  Phase  II  transfer 

•  Quantify  NCO  opportunities  to  apply  ALC  learning  in  unit 

•  Determine  11 B  ALC  Phase  II  impact  on: 

•  NCO  behaviors,  performance,  contributions,  and  standing  within  unit 

•  Squad  processes,  performance,  contributions,  and  standing  within  unit 

•  Unit  training,  operations  and  environment  (company  and  below) 

•  Examine  role  of  biographical  variables  in  ALC  transfer 

•  Explore  unit  factors  that  promote  application  of  ALC  learning 

•  Identify  Infantry  ALC  benefits  to  NCOs  and  their  units 

•  Document  ideas  for  improving  the  11 B  ALC  Phase  II  POI 

•  Generate  recommendations  for  continuing  data  collection 
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EVALUATION  DESIGN 

•  Hypothesis:  New  11 B  POI  improves  ALC  impact  in  tactical  units 

•  New  POI  defined:  final  ALC  POI,  post-pilot  (Jun  09  and  beyond) 

•  Paradigm:  multi-source  sampling  (participants  +  associates),  unit  focus 

•  Limited  longitudinal  data  collection  with  new  POI  participants 

•  Snapshot  third-party  measurement  with  informed  associates 

•  Participants:  11 B  ALC  students/graduates  +  known  associates 

•  Students/graduates  of  schoolhouse  courses  +  linked  unit  personnel 

•  Students/graduates  of  MTT  courses  +  linked  unit  personnel 

•  Evaluation  arena:  Active  Component  units  only 

•  Geographic  limits:  CONUS  (installations  with  Infantry  Battalions) 

•  Measurement  focus:  transfer  of  ALC  learning  to  job  performance 

•  Incubation  parameters: 

•  Post-graduation  time  to  apply  ALC  learning  =  5-6  mos  desired,  2  mos  minimum 

•  Pre-  +  post-ALC  acquaintanceship  for  associates  =  3-4  mos  desired,  2  mos  min 

•  Data  collection  environment:  supervised  sessions  in  school  and  unit 


EVALUATION  APPROACH 

•  Target  three-part  population  (CONUS-based): 

-  NCOs  who  graduate  from  new  1 1 B  Phase  II  POI  -  schoolhouse 

-  NCOs  who  graduate  from  new  1 1 B  Phase  II  POI  -  MTT 

-  Unit  leaders  +  peers  who  know  NCOs  before  &  after  ALC  -  associates 

•  Use  survey,  focus  group,  SJT,  and  records  mining  methods 

•  Build  multiple  perspectives  and  redundancy  into  data  collection 

•  Expect  focus  on  ALC  impact  to  reduce  self-rating  inflation 

•  Enforce  minimum  requirements  for  incubation  parameters 

•  Orchestrate  data  collection  schedule  to  minimize  number  of  site  visits 

•  Organize  research  SMEs  into  teams  for  on-site  data  collection 

•  Seek  command  emphasis  (NCOA,  FORSCOM)  to  ensure  participation 

•  Standardize  data  collection  by  means  of  Data  Collector’s  Guide 

•  Use  post-graduation  elapsed  time  as  covariate  or  sorting  variable 

•  Analyze  data  for  impact  of  new  POI  and  intervening  variables 
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SAMPLING  STRATEGY 

•  Establish  criterion-based  sample  selection  process: 

-  Select  schoolhouse  participants  from  2  classes  based  on  deployment  date 

-  Select  MTT  participants  from  3  scheduled  units  in  ARFORGEN  reset  phase 

-  Define  associates  as  participant-known  leaders  and  peers 

•  Limit  participants  to  NCOs  assigned  to  units  6+  months  from  deploying 

•  Enrol  all  eligible  participants  in  each  selected  class 

•  Limit  associates  to  1  leader  and  1  peer  per  participant 

•  Identify  pool  of  associates  to  satisfy  sample  size  goals 

•  Limit  sampling  footprint  to  units  at  primary  CONUS  installations 

•  Spread  schoolhouse  &  MTT  samples  among  light,  heavy,  &  Stryker  BCTs 

•  Set  reality-based  sample  size  goals: 

-  Schoolhouse  sample  (final)  =  250  participants  (2  classes  x  75%,  minus  attrition) 

-  MTT  sample  (final)  =  250  participants  (3  classes  x  90%,  minus  attrition) 

-  Associate  samples  (final)  =  500  ( schoolhouse )  +  500  ( MTT)  =  1000  total 

•  Assume  modest  attrition  after  6  mos  (schoolhouse  =  20%,  MTT  =  10%) 

•  Include  entire  class  in  start-of-course  and  end-of-course  surveys 


CRITICAL  ASSUMPTIONS 

•  Only  the  Phase  II  POI  for  Infantry  ALC  is  changing 

•  The  new  POI  will  remain  stable  for  at  least  12  months 

•  The  schoolhouse  and  MTT  courses  are  functionally  equivalent 

•  Schoolhouse  classes  of  -200  students  will  start  every  5-6  weeks 

•  At  least  3  MTT  classes  of  80-120  students  each  will  occur  per  quarter 

•  At  least  75%  of  schoolhouse  graduates  will  return  to  CONUS  units 

•  A  Co-  or  Pit-level  leader  can  assess  schoolhouse  +  MTT  participants 

•  Sufficient  associates  will  have  pre-  +  post-ALC  acquaintanceship 

•  Participating  units  will  allocate  time  for  data  collection  activities 

•  50%  reduction  in  desired  incubation  period  would  yield  useful  data 

•  ALC  stakeholders  will  endorse/support  execution  of  evaluation 
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MEASUREMENT  STRATEGY 

•  Use  multiple  methods  -  survey,  focus  group,  SJT,  checklist,  records 

•  Link  measures  of  11 B  ALC  impact  to  unit  training  and  operations 

•  Blend  objective,  self-report,  and  subjective  measurement  techniques 

•  Enhance  sensitivity  to  POI  effects  by  emphasizing  quantitative  measures 

•  Iterate  core  measures  across  participant  and  associate  groups 

•  Allocate  key  measures  to  two  or  more  methods  for  redundancy 

•  Superimpose  data  collection  within  operational  environment 

•  Employ  on-site  data  collectors  to  execute  or  supervise  data  collection 

•  Use  machine-scanned  questionnaire  forms  whenever  warranted 

•  Document  focus  groups  via  thematic  summaries  and  voice  recordings 

•  Track  attrition,  false  positives,  dead-ends,  and  their  causes  in  real  time 

•  Record  intervening  variables  and  confounding  factors 

•  Assign  data  manager  to  inventory  data  and  manage  central  database 

•  Take  proactive  steps  to  maintain  quality  and  quantity  of  data 

•  Protect  privacy  of  individual  participants  and  units 
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METRICS  CATEGORIES 

•  Critical  and  Creative  Thinker 

•  Warrior  Leader 

•  Military  Leadership 

•  Warrior  Competencies 

•  Leader  Developer 

•  Counseling,  Coaching,  and  Mentoring 

•  Training  Subordinates 

•  Shaping  Unit  Performance 

•  Expanding  Own  Competencies 

•  Ambassador 

•  All  Environments 

•  Operational  Environment 

•  Resource  Manager 

•  Global  Impact 
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TYPES  OF  MEASURES  (PRIMARY) 

•  Extent  of  ALC  practice/opportunities/learning  (nco  ratings) 

•  NCO’s  opportunities  to  apply  ALC  learning  in  unit  (nco  checklist,  Ldr  estimates) 

•  NCO’s  level  of  proficiency  on  competencies  (NCO  ratings  pre-  and  post-training,  SJTs) 

•  Extent  of  perceived  transfer  of  ALC  learning  (nco  ratings) 

•  Extent  of  ALC  impact  on  NCO’s  performance,  KSAs,  and  competencies 

•  Ratings  by  NCOs,  unit  leaders  and  peers 

•  Comments  by  NCOs,  unit  leaders  and  peers 

•  Extent  of  ALC  impact  on  squad’s  performance  and  processes 

•  Ratings  by  NCOs,  unit  leaders  and  peers,  CTC  O/Cs 

•  Comments  by  NCOs,  unit  leaders  and  peers 

•  Extent  of  positive  feedback  re:  ALC  impact  (ratings  and  comments  by  NCOs) 

•  Listing  of  NCO  activities/actions/events/contacts  (provided  by  ncos,  unit  leaders) 

•  Examples  of  performance/competency/process  impact  for  NCO  or  squad 

•  Comments  by  NCOs,  unit  leaders  and  peers 

•  Listing  Of  transfer-relevant  events  in  unit  (e.g.,  new  equipment  received  by  unit) 

•  Comments  by  NCOs,  unit  leaders  and  peers 

•  Performance  scores  for  NCO,  squad  members,  or  squad  (from  unit  records) 

•  Incidents  related  to  ALC  impact  (from  unit  records) 


TYPES  OF  MEASURES  (GLOBAL) 

•  Relevance  of  ALC  training  to  NCO  job  performance  (nco  ratings) 

•  Extent  of  ALC  impact  on  NCO’s  performance  outcomes  (e  g.,  contributions) 

•  Ratings  by  NCOs,  unit  leaders  and  peers 

•  Comments  by  NCOs,  unit  leaders  and  peers 

•  Extent  of  ALC  impact  on  NCO’s  self-image  (e.g.,  commitment  to  Army) 

•  Ratings  and  comments  by  NCOs 

•  Extent  of  ALC  impact  on  NCO’s  or  squad’s  standing  in  the  unit 

•  Ratings  by  NCOs,  unit  leaders  and  peers 

•  Comments  by  NCOs,  unit  leaders  and  peers 

•  Extent  of  NCO’s  impact  on  unit  training  and  operations  (ALC-reiated) 

•  Ratings  by  NCOs,  unit  leaders  and  peers 

•  Comments  by  NCOs,  unit  leaders  and  peers 

•  Unit  environment  as  it  affects  NCO’s  use  of  ALC  learning  (e.g.,  time,  workload) 

•  Ratings  by  NCOs,  unit  peers 

•  Comments  by  NCOs,  unit  peers 

•  ALC  benefits  to  NCO  and  squad 

•  Comments  by  NCOs,  unit  leaders  and  peers 
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DATA  COLLECTION  INSTRUMENTS 


•  Survey  Questionnaires 

-  Biographical  Inventory  (integrated  with  primary  questionnaire  as  appropriate) 

-  ALC  Students  Survey  (start-of-course  and  end-of-course  versions) 

-  Participants  Survey  (end-state) 

-  Unit  Leaders  Survey  (including  section  for  Platoon  Sergeants  only) 

-  Unit  Peers  Survey 

•  Focus  Group  Protocols 

-  ALC  Students  Focus  Group  (end-of-course) 

-  Participants  Focus  Group  (end-state) 

-  Unit  Leaders  Focus  Group 

-  Unit  Peers  Focus  Group 

•  Miscellaneous  Instruments 

-  Situational  Judgment  Tests  (start-of-course  and  end-of-course  versions) 

-  Application  Checklist  (recurring) 

-  Unit  Records-Based  Capture  Form 

-  Master  Data  Inventory 
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DATA  SOURCES 

•  ALC  students/graduates  -  squad  leaders  for  at  least  2  months 

•  Unit  leaders  -  Pit  Sgt,  Pit  Ldr,  Co  1SG,  Co  Cdr  (within  NCO’s  Co) 

•  Unit  peers  -  other  squad  leaders  (within  NCO’s  Co) 

•  CTC  O/Cs  -  via  end-of-rotation  take  home  packages 

•  Unit  performance  records  -  routine  testing  and  training  records,  etc. 

•  Unit  incident  records  -  personnel  reports,  UCMJ  records,  etc. 

•  Data  collectors  -  on-site  researchers  (significant  events,  exceptions) 

•  Investigators  -  scientists,  SMEs,  etc.  (lessons  learned) 


F-8 


DATA  COLLECTION  STAGES 


•  Start-of-ALC  Snapshot  ( schoolhouse  and  MTT  participants) 

-  Biographical  inventory  (all)  +  survey  (all)  +  SJT  (all) 

•  End-of-ALC  Snapshot  ( schoolhouse  and  MTT  participants) 

-  Survey  (all)  +  SJT  (all)  +  focus  group  (subset) 

•  Application  Period  Tracking  ( schoolhouse  and  MTT  participants) 

-  ALC-related  activity  checklist  (all,  by  pay  period) 

•  End  State  Assessment 

-  Participants  ( Schoolhouse  and  MTT ):  Survey  (all)  +  focus  group  (subset) 

-  Unit  Leaders:  Survey  (all)  +  focus  group  (subset) 

-  Unit  Peers:  Survey  (all)  +  focus  group  (subset) 

-  On-site  data  collectors:  Records-based  data  +  confounding  factors 
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DATA  COLLECTION  SCHEDULE 
(STRAWMAN) 
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ADMINISTRATIVE  NOTES 


A.  This  plan  will  require  updating  as  part  of  the  follow-on  preparations  for  data  collection. 

B.  This  plan  may  require  periodic  updating  during  actual  data  collection  to  incorporate 
changes  in  data  collection  procedures  and/or  materials. 

C.  The  Data  Manager  is  responsible  for  maintaining  and  distributing  this  plan. 

D.  Suggestions  for  modifying  this  plan  should  be  submitted  to  the  point  of  contact  below. 

E.  Point  of  contact  for  this  plan  is  Dr.  Robert  Pleban,  (706)  545-2207, 
Robert.Pleban@us.army.mil. 
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Infantry  ALC  Training  Transfer  Evaluation 
DATA  COLLECTION  AND  MANAGEMENT  PLAN  (DCMP) 

ARI  -  Fort  Benning  Research  Unit 


Scope  of  the  Plan 

This  document  lays  out  the  plans  to  accomplish  a  100%  data  collection 
effort  for  the  targeted  sample  of  Soldiers.  The  plan  is  flexible  so  that 
specific  features  may  be  rescaled  to  reduce  resource  requirements  or 
troop  support  burden. 


1 .  References 

a.  Northrop  Grumman  Technical  Services,  FarXfer  Data  Collection  Instruments  -  Outline  Design 
(Version  3.0),  2  Mar  09. 

b.  Northrop  Grumman  Technical  Services,  ALC  Training  Transfer  Evaluation  Design  Plan  (Final),  27 
Feb  09. 

c.  Northrop  Grumman  Technical  Services,  ALC  Far  Transfer:  Master  List  of  Measures  (Version  3.0), 
26  Feb  09. 

d.  Northrop  Grumman  Technical  Services,  Candidate  Metrics  of  BNCOC/ALC  Impact  (Version  6.2), 
delivered  15  Sep  08. 

e.  Northrop  Grumman  Technical  Services,  Comprehensive  Work  Plan  (Final)  -  BNCOC  Far  Transfer, 
14  Apr  08. 

f.  U.S.  Army  Research  Institute,  Development  and  Methodology  for  the  Administration  of  a  Basic  Non¬ 
commissioned  Officer  Course  (BNCOC)  Survey  of  Far  Transfer  -  Soldier  Readiness  (Statement  of 
Work),  issued  14  Feb  08. 

2.  Key  Personnel  Defined 

a.  Tactical  Units 

(1)  Participant —  1 1B  ALC  graduate  (new  Phase  II  POI)  who  serves  as  a  squad  leader. 

(2)  Associates  —  Unit  leaders  and  peers  who  know  a  participant  before  and  after  ALC. 

(3)  Unit  leader —  Representative  leader  of  a  participant,  within  the  same  company. 

(4)  Unit  peer —  Representative  peer  (squad  leader)  of  a  participant,  within  the  same  company. 

(5)  Cohort  —  A  group  of  participants  from  the  same  ALC  class,  plus  their  associates. 

b.  Research  Team 

(1)  Evaluation  manager  —  ARI’s  principal  investigator  who  directs  the  evaluation. 

(2)  Data  manager —  ARI’s  investigator  who  manages  data  operations. 

(3)  Data  collectors —  Researchers  who  conduct,  supervise  or  monitor  data  collection  events. 

(4)  Facilitators  —  Data  collectors  (SMEs)  who  lead  interactive  data  collection  sessions. 
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3.  Sampling  Procedures 

a.  Classes  Targeted.  The  sample  pool  is  defined  by  the  class  schedule  following  lock-in  of  the  new 
POI.  Classes  scheduled  in  the  schoolhouse  (NCO  Academy,  Fort  Benning)  and  at  home  stations 
(Mobile  Training  Team,  or  MTT)  are  targeted,  approximately  6  months  after  lock-in  of  the  new  POI. 

(1)  Schoolhouse  classes,  2  total  (parent  units  =  light,  heavy,  and  Stryker  BCTs) 

(2)  MTT  classes  in  CONUS,  1  each  for  light,  heavy,  and  Stryker  BCTs  in  reset  phase  (3  total) 

b.  Selecting  Participants.  The  following  criteria  will  govern  selection  of  participants : 

(1)  Schoolhouse  classes:  NCOs  must  be  squad  leaders  remaining  with  same  CONUS  units  and  2- 
6  months  from  deploying  (5-6  months  desired);  starting  sample  =  all  eligible  students  (-75%  of 
class,  or  120-150  NCOs);  final  sample  =  80%  of  starting  sample  (assuming  20%  attrition)  or 
less  if  dictated  by  resource  constraints 

(2)  MTT  classes  (units  in  reset  phase):  NCOs  must  be  squad  leaders  scheduled  to  remain  with 
unit  at  least  2-6  months  (5-6  months  desired);  starting  sample  =  all  eligible  students  (-90%  of 
class,  or  55-90  NCOs);  final  sample  =  90%  of  starting  sample  (assuming  10%  attrition)  or  less  if 
dictated  by  resource  constraints 

c.  Selecting  Associates.  The  following  criteria  will  govern  selection  of  associates : 

(1)  Unit  Leaders  (1  per  participant):  supervisory  personnel  listed  by  participant,  who  directly 
observe  the  participant  at  least  2-4  months  before  and  2-4  months  after  ALC,  located  in 
CONUS  at  time  of  data  collection 

(2)  Unit  Peers  (1  per  participant):  fellow  squad  leaders  listed  by  participant,  who  directly  observe 
the  participant  at  least  2-4  months  before  and  2-4  months  after  ALC,  located  in  CONUS  at  time 
of  data  collection,  and  preferably  not  serving  as  a  participant 

d.  Rostering.  Based  on  the  initial  information  (start-of-course  survey)  gathered  from  students  in  the 
targeted  classes,  the  data  manager  will  screen  out  ineligible  candidates  and  prepare  a  master  list  of 
participants,  leaders,  and  peers. 

e.  Non-Overlap  between  Participants  and  Peers.  When  rostering  peers,  the  data  manager  will  avoid 
selecting  an  individual  who  is  also  a  participant,  given  a  reasonable  choice.  This  is  meant  to  avoid 
peers  comparing  themselves  (competing)  with  participants  whom  they  rate  in  surveys. 

f.  Expected  Attrition. 

(1)  10-20%  attrition  of  MTT  and  schoolhouse  participants 

(2)  Up  to  25%  attrition  of  associates 

g.  Close-out  Authority.  The  data  manager  will  assess  sampling  outcomes  and  verify  the  sufficiency  of 
each  sub-sample,  with  the  evaluation  manager’s  concurrence. 

4.  Data  Requirements 

a.  Primary  Requirements.  A  comprehensive  list  of  primary  measures  appears  in  Reference  1c.  The 
measures  fall  under  the  five  roles  of  a  multifunctional  NCO:  critical  and  creative  thinker,  warrior 
leader,  leader  developer,  ambassador,  and  resource  manager. 

b.  Supplemental  Requirements.  Secondary  measures  address  participant  attributes,  training  and 
application  opportunities,  unit  training  events,  unit  environment,  and  global  impact. 

c.  Data  Sources'.  The  complete  family  of  data  requirements  targets  the  following  data  sources: 

(1)  Participants  (NCOs  during  and  after  ALC  attendance) 

(2)  Associates  (unit  leaders  and  peers  who  know  a  participant  before  and  after  ALC) 

(3)  Take  home  packages  from  CTC  rotations 
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(4)  Unit  records  (training  schedules,  testing  records,  personnel  reports,  UCMJ  records,  etc.) 

d.  Data  Tagging.  Critical  data  tags  include  Soldier  identification  number  (used  for  all  components  of  a 
record),  group  (schoolhouse  vs.  MTT),  role  (participant,  unit  leader,  unit  peer),  unique  unit  identifier 
(down  to  squad),  installation,  and  date-time  when  data  were  collected. 

5.  Data  Collection  Instruments 

a.  Master  List.  Reference  la  outlines  the  data  collection  instruments,  their  design  parameters,  and  the 
scope  of  their  contents. 

b.  Instrument  Types.  Five  kinds  of  data  collection  instruments  are  used:  survey  questionnaires,  focus 
group  protocols,  situational  judgment  tests,  participant  checklists,  and  forms  for  capturing  data  from 
unit  records.  The  intended  medium  is  hardcopy. 

c.  Survey  Questionnaires. 

•  Biographical  Inventory  (integrated  with  primary  questionnaire  where  feasible) 

•  ALC  Students  Survey  (start-of-course  and  end-of-course  versions) 

•  Participants  Survey  (end-state) 

•  Unit  Leaders  Survey  (including  section  for  Platoon  Sergeants  only) 

•  Unit  Peers  Survey 

d.  Focus  Group  Protocols. 

•ALC  Students  Focus  Group  (end  of  course) 

•  Participants  Focus  Group  (end  state) 

•  Unit  Leaders  Focus  Group 

•  Unit  Peers  Focus  Group 

e.  Miscellaneous  Instruments. 

•  Situational  Judgment  Tests,  or  SJTs  (start-of-course  and  end-of-course  versions) 

•Application  Checklist  (recurring) 

•  Unit  Records-Based  Capture  Form 

•  Master  Data  Inventory 

f.  External  Approval.  Instruments  that  produce  a  record  of  Soldier  responses  (questionnaires,  focus 
group  protocols,  SJTs,  checklists)  will  be  submitted  for  verification  of  exempt  status. 

g.  Machine  Scan  Capability.  Instruments  that  record  Soldier  responses  (questionnaires,  SJTs, 
checklists)  will  be  prepared  in  machine  scanned  (mark  sense)  form  whenever  feasible. 

h.  Production.  The  data  manager’s  office  will  produce  hardcopies  of  Soldier-completed  forms 
(questionnaires,  SJTs,  checklists)  and  ship  them  to  the  data  collection  sites.  Other  forms  (focus 
group  protocols,  records-based  capture  form)  may  be  produced  locally.  The  central  production  of 
key  forms  will  reduce  problems  related  to  machine  scanning  and  fractionated  quality  control. 

i .  Approval  of  Changes. 

(1)  The  use  of  standard  data  collection  instruments,  as  approved  by  the  evaluation  manager,  is 
essential  to  the  collection  of  high  quality  data. 

(2)  The  evaluation  manager  must  approve  any  modifications  or  adaptations  of  the  data  collection 
instruments. 

6.  Data  Collection  Procedures 

a.  General. 

(1)  Standardization.  Standard  procedures  will  help  ensure  the  quality,  integrity,  and  privacy  of  the 
multi-faceted  data.  Members  of  the  research  team  will  not  depart  from  the  established 
procedures  without  approval  from  the  evaluation  manager,  unless  it  is  unavoidable. 
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(2)  Investigative  control.  The  data  manager  will  direct  and  monitor  all  data  collection  activities. 

The  evaluation  manager  must  approve  all  planned  departures  from  standard  procedures. 

(3)  Data  Collection  Teams.  Fully  qualified  and  trained  data  collectors  will  conduct,  supervise,  or 
monitor  the  collection  of  data.  Data  collectors  will  work  in  two-person  teams.  One  of  the 
members  must  be  a  military  SME.  As  a  general  rule,  both  members  of  a  team  will  be  needed 
for  data  collection  sessions,  except  for  records-based  capture  sessions. 

(4)  Data  Collector’s  Guide.  All  data  collectors,  including  session  facilitators,  will  use  a  job  aid  in 
the  form  of  a  common  Data  Collector’s  Guide.  The  user-friendly  guide  will  be  a  key  tool  for 
standardizing  the  data  collection  procedures. 

(5)  Data  Collection  Sites.  Data  collection  activities  will  take  place  at  schoolhouse  or  unit  sites  (see 
para  1 1  below).  Data  collection  teams  will  work  on  site  using  Government  facilities. 

(6)  Site  Visits.  Data  collectors  will  travel  to  schoolhouse  and  unit  sites  to  collect  data.  Site  visits 
will  be  planned  to  maximize  the  data  collection  opportunities  during  each  visit  and  to  minimize 
the  number  of  visits  to  each  schoolhouse  and  installation. 

(7)  Procedural  Documentation.  Data  collectors  will  document  all  planned  and  unplanned 
departures  from  the  standard  procedures.  The  documentation  will  be  submitted  to  the  data 
manager  to  become  part  of  the  database. 

(8)  Confidentiality  of  Data.  Protecting  the  privacy  of  Soldiers  and  units  is  an  absolute  obligation  of 
the  research  team.  All  hardcopy  and  electronic  data  must  be  treated  as  confidential  and 
protected  against  access  by  anyone  outside  ARI’s  research  team. 

b.  Survey  (Questionnaire)  Sessions. 

(1)  Administration.  A  data  collection  team  will  administer  survey  sessions,  preparing  in  advance 
and  managing  each  session.  Surveys  will  be  administered  to  groups  as  large  as  200  Soldiers. 
A  team  will  handle  only  one  survey  group  at  a  time,  unless  simultaneous  groups  are  located  in 
close  proximity.  For  groups  smaller  than  25  Soldiers,  only  one  data  collector  should  suffice. 

(2)  Advance  Preparation.  The  lead  data  collector  will  accomplish  final  coordination  of  the  schedule 
and  location  for  each  survey  session.  S/he  will  organize  hardcopies  of  all  questionnaire  forms 
(received  from  the  central  office)  needed  for  each  session,  and  fill  in  any  fields  reserved  for  use 
by  the  research  team.  S/he  will  also  obtain  pencils  or  pens  to  take  to  the  session. 

(3)  During  the  Session.  At  the  start  of  the  session  the  data  collection  team  will  introduce  the 
survey,  then  distribute  the  forms,  provide  pencils/pens  as  needed,  and  answer  questions. 

(4)  End  of  Session.  The  data  collectors  will  gather  the  forms  completed  by  the  Soldiers.  They  will 
verify  the  completeness  of  the  administrative  information  on  each  questionnaire  before 
releasing  the  individuals  as  they  finish.  When  a  focus  group  follows,  the  Soldiers  will  be  asked 
to  remain  in  the  area  for  the  follow-on  activity. 

(5)  After  the  Session.  The  lead  data  collector  will  execute  the  procedures  for  submitting  and 
storing  data  specified  in  paragraphs  6e  and  6f  below. 

c.  Focus  Group  Sessions. 

(1)  Data  Collection  Team.  When  conducting  focus  groups,  the  members  of  a  data  collection  team 
will  fill  two  roles — facilitator  and  note  taker.  The  facilitator  will  be  a  military  SME,  preferably 
with  previous  facilitation  experience.  The  note  taker  should  be  familiar  with  military  operations 
and  terms. 

(2)  Group  Composition.  A  focus  group  session  will  involve  a  group  ranging  from  3  to  6  Soldiers  (4- 
6  participants,  3-5  unit  leaders ,  4-6  unit  peers).  A  focus  group  will  contain  Soldiers  of  only  one 
type  (participants  or  leaders  or  peers).  For  example,  leaders  and  peers  will  not  participate  in 
the  same  group,  to  avoid  curtailing  full  and  frank  participation. 
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(3)  Administration.  The  facilitator  will  handle  administrative  aspects  of  focus  groups,  preparing  in 
advance  and  managing  the  session  itself.  As  a  general  rule,  a  facilitator  will  conduct  no  more 
than  four  focus  group  sessions  in  a  day. 

(4)  Advance  Preparation.  The  data  collectors  will  conduct  final  coordination  of  the  schedule  and 
location  for  each  focus  group  session.  Working  with  the  appropriate  focus  group  protocol,  the 
facilitator  will  prioritize  the  questions  and  tailor  the  protocol  for  the  specific  group,  annotating 
the  protocol  in  the  process. 

(5)  During  the  Session.  The  facilitator  will  use  the  annotated  protocol  as  a  guide  for  each  group  or 
individual  session.  The  facilitator  will  encourage  interaction  among  the  Soldiers  in  the  group, 
fostering  a  collaborative  brainstorming  environment.  The  focus  group  protocol  will  specify 
procedures  for  eliciting  insights  on  issues  of  interest.  The  note  taker  will  capture  the  key  points 
of  the  dialogue,  using  a  laptop  computer.  As  a  general  rule,  voice  dialogue  will  be  digitally 
recorded  for  later  reference. 

(6)  After  the  Session.  After  releasing  the  Soldiers,  the  data  collectors  will  transfer  the  voice  file  to 
a  digital  storage  medium  (e.g.,  laptop  computer  or  flash  drive),  using  a  standard  format  for 
naming  each  file.  At  the  end  of  each  day,  the  note  taker  will  flesh  out  the  notes  in  a  Microsoft 
Word®  file,  and  the  facilitator  will  review  the  notes  before  they  are  finalized.  A  designated  data 
collector  will  execute  the  procedures  for  submitting  and  storing  data  specified  in  paragraphs  6e 
and  6f  below. 

d.  Other  Data  Collection  Activities. 

(1)  SJT  Sessions. 

(a)  Administration.  A  data  collection  team  will  administer  each  SJT  session,  preparing  in 
advance  and  managing  the  session  itself.  SJTs  will  be  administered  to  groups  as  large  as 
200  Soldiers.  A  team  will  handle  only  one  SJT  group  at  a  time. 

(b)  Advance  Preparation.  The  lead  data  collector  will  accomplish  final  coordination  of  the 
schedule  and  location  for  each  SJT  session.  S/he  will  ready  the  hardcopy  SJT  forms 
(received  from  the  central  office)  for  each  session,  and  fill  in  any  fields  reserved  for  use  by 
the  research  team.  S/he  will  also  obtain  pencils  or  pens  to  take  to  the  session. 

(c)  During  the  Session.  At  the  start  of  the  session  the  data  collection  team  will  introduce  the 
SJT,  then  distribute  the  forms,  provide  pencils/pens  as  needed,  answer  questions,  and 
enforce  time  limits  as  applicable. 

(d)  End  of  Session.  The  data  collectors  will  gather  the  forms  completed  by  the  Soldiers.  They 
will  verify  the  completeness  of  the  administrative  information  on  each  SJT  form  before 
releasing  the  students  as  they  finish. 

(e)  After  the  Session.  The  lead  data  collector  will  execute  the  procedures  for  submitting  and 
storing  data  specified  in  paragraphs  6e  and  6f  below. 

(2)  Participant  Checklist. 

(a)  Basic  Process.  During  the  application  period  following  ALC  graduation,  each  participant  will 
complete  a  checklist  twice  each  month  to  record  the  actual  application  of  ALC-acquired 
knowledge  and  skills. 

(b)  Distribution  of  Forms.  The  data  manager  will  mail  hardcopies  of  the  checklist  to  a  unit  POC 
to  distribute  locally.  If  feasible,  the  data  manager  may  email  the  checklist  file  to  each 
participant,  who  will  print  the  form  or  use  it  digitally.  As  a  fallback  option,  the  data  manager 
may  mail  a  3-month  supply  of  printed  forms  to  a  participant. 

(c)  Recording  Cycle.  A  participant  will  record  application  data  for  intervals  of  about  two  weeks, 
with  the  cycle  concluding  at  the  end  of  a  pay  period. 
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(d)  Reminders.  The  data  manager  will  send  regular  reminders  (semi-monthly)  to  each 
participant,  via  email  or  regular  mail.  Despite  the  increased  administrative  workload,  the 
prospects  of  avoiding  very  low  return  rates  should  justify  the  extra  effort. 

(3)  Capture  of  Records-Based  Data. 

(a)  Basic  Process.  A  data  collector  will  use  the  Records-Based  Capture  Form  to  extract 
specified  data  elements  from  unit  records  during  an  on-site  visit. 

(b)  Advance  Preparation.  The  data  collector  will  schedule  one  or  more  sessions  in  the 
appropriate  battalion  and  company  HQ.  S/he  will  become  thoroughly  familiar  with  the 
capture  form  prior  to  the  first  session. 

(c)  Data  Capture.  At  the  start  of  each  session  the  data  collector  will  brief  the  unit  POC  on  the 
purpose  of  the  data  capture.  The  data  collector  will  locate  each  specified  data  element  in 
the  appropriate  unit  record(s)  and  manually  record  the  element  on  the  capture  form. 
Alternatively,  a  designated  unit  POC  may  provide  the  required  data  elements  for  the  data 
collector  to  record  on  the  capture  form. 

(d)  Cautions.  The  data  collector  will  comply  with  all  rules  and  provisions  specified  by  the  unit 
POC.  S/he  will  not  photocopy  or  remove  any  portion  of  unit  records,  nor  will  s/he  make 
marks  or  annotations  on  any  unit  records. 

e.  Submission  of  Data. 

(1)  Standard  Forms  (non-recurring).  The  lead  data  collector  will  deliver  completed  surveys,  SJT 
forms,  and  records-based  capture  forms  in  person  to  the  data  manager.  If  original  forms  are 
shipped  to  the  data  manager,  a  backup  copy  of  each  item  will  be  made  and  stored. 

(2)  Focus  Group  Records.  For  each  completed  focus  group,  a  designated  data  collector  will  send 
a  digital  copy  of  the  compiled  notes  (and  voice  recording,  if  directed)  to  the  data  manager. 

(3)  Participant  Checklists  (recurring).  At  the  end  of  every  pay  period  each  participant  will  submit 
his/her  latest  checklist  to  the  unit  POC,  who  will  forward  the  forms  to  the  data  manager  via 
email  attachment  or  fax  or  regular  mail. 

(4)  Documentation  of  Exceptions.  Data  collectors  will  submit  to  the  data  manager  written 
documentation  of  all  planned  and  unplanned  departures  from  standard  procedures. 

f.  Local  Protection  of  Data. 

(1)  Local  Storage.  Backup  copies  of  data  in  hardcopy  and  digital  forms  will  be  maintained  by  each 
data  collection  team.  They  will  not  be  destroyed  unless  the  data  manager  so  directs. 

(2)  Protection  of  Data.  All  hardcopy  and  digital  materials  containing  Soldier  information,  including 
digital  files  and  focus  group  notes,  will  be  stored  under  controlled  access  conditions,  with  no 
access  by  personnel  outside  the  research  team. 

(3)  Records-Based  Data.  All  pages  of  the  Records-Based  Capture  Form  will  be  marked  “FOUO.” 
Locally  maintained  copies  of  these  forms  will  be  stored  in  an  enclosed  folder  or  container 
prominently  marked  “FOUO.” 

7.  Data  Collection  Schedule 

a.  Basics.  The  data  collection  period  is  expected  to  run  from  January  2010  through  October  2010,  with 
cohorts  staggered  by  the  date  of  the  selected  courses.  Table  1  outlines  the  four  phases  of  data 
collection  for  a  given  cohort.  The  application  period  after  ALC  graduation  will  be  enforced  to  allow 
participants  sufficient  time  to  apply  their  ALC  learning  (5-6  months  desired ,  2  months  minimum).  An 
acquaintanceship  period  during  which  associates  observe  a  participant  will  be  enforced  (at  least  4 
months  desired  before  and  after  ALC,  2  months  minimum  before  and  after  ALC). 
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Table  1.  Phases  of  Data  Collection 


Phase 

Desired  Timeframe 

Primary  Data  Collection 

Course-Start  Snapshot 

First  2  days  of  course 

Biographical  Inventory,  Student  Survey,  SJT 

Course-End  Snapshot 

Final  3  days  of  course 

Student  Survey,  SJT,  Student  Focus  Group 

Application/Incubation 

5-6  months  working  in  unit 

Tracking,  Participant  Checklist 

End  State  Assessment 

5-6  months  post-ALC 

Participant  Survey  +  Focus  Group,  Associate 
Survey  +  Focus  Group,  Records  Mining 

b.  Scheduling.  The  research  team  will  coordinate  with  schoolhouse,  MTT,  and  unit  POCs  to  create  a 
cohort-specific  data  collection  schedule  beginning  with  the  start  of  the  course  and  ending  ~7  months 
later.  Table  2  gives  the  scheduling  parameters  of  the  events  planned  for  one  schoolhouse  cohort 
(-200  students,  -125  final  participants,  -250  associates).  Table  3  gives  the  scheduling  parameters 
for  one  MTT  cohort  (-100  students,  -80  final  participants,  -160  associates).  Scheduling  end  state 
events  for  a  schoolhouse  cohort  will  be  more  challenging  compared  to  an  MTT  cohort,  because 
graduates  of  a  schoolhouse  course  will  return  to  geographically  scattered  units. 


Table  2.  Scheduling  Parameters  for  a  Schoolhouse  Cohort 


Event 

When 

Duration 

# 

Soldiers 

Grp 

Size 

Est# 

Grps 

Collectors 
per  Group 

Start  and  End  of  Course  Snapshots*  (occurring  at  Fort  Benning) 

Student  Survey,  Start-of-Course 

Day  1  or  2 

1  hour 

-200 

-200 

1 

2 

Situational  Judgment  Test  #1 

Day  1  or  2 

1  hour 

-200 

-200 

1 

2 

Student  Survey,  End-of-Course 

Days  25-26 

14  hour 

-200 

-200 

1 

2 

Situational  Judgment  Test  #2 

Days  25-26 

1  hour 

-200 

-200 

1 

2 

Student  Focus  Gp  (End-of-Course) 

Days  26-27 

2  hours 

25-30 

5-6 

5 

2 

End  State  Assessment*  (distributed  across  6  installations) 

Participant  Survey 

Graduation  +  5-6  mos 

1  hour 

125 

9-12 

12 

1 

Unit  Leader  Survey 

Graduation  +  5-6  mos 

14  hour 

125 

9-12 

12** 

1 

Unit  Peer  Survey 

Graduation  +  5-6  mos 

14  hour 

125 

9-12 

12** 

1 

Participant  Focus  Group 

Graduation  +  5-6  mos 

2  hours 

125 

4-6 

24*** 

2 

Unit  Leader  Focus  Group 

Graduation  +  5-6  mos 

114  hours 

125 

3-5 

30*** 

2 

Unit  Peer  Focus  Group 

Graduation  +  5-6  mos 

114  hours 

125 

4-6 

24*** 

2 

Records-Based  Capture  Session 

Graduation  +  6  mos 

1-2  hours 

N/A 

N/A 

20-30 

1 

*  Only  one  data  collection  team  will  work  at  each  site  during  course  snapshot  and  end  state  phases. 

**  Survey  and  focus  group  sessions  should  be  scheduled  together  for  leaders  and  peers. 

***  Based  on  four  or  five  focus  groups  per  site  (installation)  visit. 

Assumption'.  At  end  state,  cohort  members  are  more  or  less  evenly  distributed  across  6  installations. 

c.  Responsibilities.  The  data  manager  will  prepare  the  complete  schedule  for  each  schoolhouse 
cohort,  with  input  from  the  schoolhouse  data  collection  team  leader.  For  a  given  MTT  cohort, 
located  at  one  installation,  the  primary  data  collection  team  leader  may  prepare  the  schedule  (to 
capitalize  on  local  cognizance),  in  coordination  with  the  data  manager.  The  data  manager  will 
maintain  a  master  schedule  that  integrates  data  collection  requirements  and  personnel  across 
installations  and  cohorts.  The  data  manager  will  direct  internal  coordination  of  schedules  to  ensure 
team-wide  awareness  and  resolve  scheduling  conflicts  or  issues.  The  primary  data  collection  team 
leader  will  coordinate  with  local  schoolhouse,  MTT,  and  unit  POCs  whenever  possible.  The 
evaluation  manager  will  resolve  significant  schedule  conflicts  and  issues. 
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Table  3.  Scheduling  Parameters  for  an  MTT  Cohort 


Event 

When 

Duration 

# 

Soldiers 

Grp 

Size 

Est# 

Grps 

Collectors 
per  Group 

Start  and  End  of  Course  Snapshots  (occurring  at  one  installation) 

Student  Survey,  Start-of-Course 

Day  1  or  2 

1  hour 

-100 

-100 

1 

2 

Situational  Judgment  Test  #1 

Day  1  or  2 

1  hour 

-100 

-100 

1 

2 

Student  Survey,  End-of-Course 

Days  25-26 

!4  hour 

-100 

-100 

1 

2 

Situational  Judgment  Test  #2 

Days  25-26 

1  hour 

-100 

-100 

1 

2 

Student  Focus  Gp  (End-of-Course) 

Days  26-27 

2  hours 

25-30 

5-6 

5 

2 

End  State  Assessment*  (occurring  at  one  installation) 

Participant  Survey 

Graduation  +  5-6  mos 

1  hour 

80 

40 

2 

2 

Unit  Leader  Survey 

Graduation  +  5-6  mos 

!4  hour 

80 

6-7 

12** 

1 

Unit  Peer  Survey 

Graduation  +  5-6  mos 

!4  hour 

80 

6-7 

12** 

1 

Participant  Focus  Group 

Graduation  +  5-6  mos 

2  hours 

60*** 

4-6 

12 

2 

Unit  Leader  Focus  Group 

Graduation  +  5-6  mos 

VA  hours 

50*** 

3-5 

12 

2 

Unit  Peer  Focus  Group 

Graduation  +  5-6  mos 

VA  hours 

60*** 

4-6 

12 

2 

Records-Based  Capture  Session 

Graduation  +  6  mos 

1-2  hours 

N/A 

N/A 

20-30 

1 

*  Two  data  collection  teams  will  work  on-site  during  the  end  state  phase. 

**  Survey  and  focus  group  sessions  should  be  scheduled  together  for  leaders  and  peers,  with  excess 
individuals  excused  from  focus  group  duty  as  necessary. 

***  Focus  group  sampling  is  limited  to  60-75%  due  to  time  constraints  and  group  size  limits. 

Assumption'.  At  end  state,  cohort  members  are  located  at  the  same  installation. 

d.  Constraints.  To  minimize  the  intrusion  factor  for  tactical  units,  it  is  highly  desirable  to  schedule  all 
data  collection  events  for  a  BCT  within  a  five-day  week.  In  the  case  of  a  schoolhouse  cohort,  the 
number  of  Soldier-in-the-loop  end  state  events  at  a  given  installation  (less  than  20,  per  Table  2) 
should  fit  comfortably  within  a  week.  In  the  case  of  an  MTT  cohort,  however,  all  end  state  events 
will  occur  at  the  same  installation  and  the  events  could  total  40  in  number  (see  Appendix  G  - 
Notional  Schedule  for  MTT  Cohort).  If  a  data  collection  team  is  able  to  handle  up  to  20  events  in  5 
days,  scheduling  about  40  events  in  a  work  week  would  require  2  data  collection  teams  on  site  at 
the  same  time.  Thus,  availability  of  data  collection  teams  could  become  a  constraint.  Other 
constraints  will  include  unit  training  events,  holidays  with  a  short  work  week,  Family  time  for 
Soldiers,  and  perhaps  a  unit  preference  against  scheduling  a  given  Soldier  for  more  than  one 
session. 

e.  By-Name  Tasking.  Course  snapshot  events  will  cover  100%  of  enrolled  students  assembled  in  one 
large  group,  except  for  end-of-course  focus  groups  scheduled  for  20-25%  of  the  class  in  small 
groups.  The  latter  and  all  other  events  must  be  scheduled  for  specific  groups  by  name.  The  data 
collection  team  leader  will  use  the  data  manager’s  master  list  of  participants  and  associates  to 
determine  the  members  of  each  group,  and  the  by-name  listings  will  become  a  critical  part  of  the 
paperwork  to  be  coordinated  with  the  schoolhouse,  MTT,  or  unit  POCs. 

f.  Leeway.  To  accommodate  course  and  unit  constraints,  scheduling  of  data  collection  activities  may 
vary  so  long  as  the  minimum  periods  specified  above  (7a)  are  protected. 

g.  Notional  Schedule.  A  notional  schedule  for  an  MTT  cohort  appears  in  Appendix  G  (Notional 
Schedule  for  MTT  Cohort).  The  table  represents  the  start-to-finish  data  collection  activities 
associated  with  an  entire  MTT  class.  It  incorporates  the  parameters  proposed  throughout  this 
section,  including  a  1-week  window  with  2  data  collection  teams.  The  schedule  for  a  schoolhouse 
cohort  would  be  more  complicated  because  multiple  installations  would  be  involved  in  end  state 
data  collection. 
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h.  Illustrative  Timeline.  At  a  given  installation,  the  more  demanding  data  collection  schedule  is  likely  to 
be  generated  by  an  MTT  cohort  because  all  participants  and  associates  will  reside  at  that  location. 
To  convey  a  better  picture  of  the  flow  and  density  of  data  collection  events,  Table  4  presents  an 
illustrative  timeline  for  a  high-density  week  during  end  state  data  collection  with  an  MTT  cohort, 
based  on  Appendix  G  (Notional  Schedule  for  MTT  Cohort).  As  the  table  shows,  two  data  collection 
teams  would  work  at  the  same  site  to  handle  the  one-week  window  targeted  for  the  convenience  of 
the  units. 


Table  4.  One-Week  Timeline  for  End  State  Events  with  an  MTT  Cohort 


Day 

Morning 

Afternoon 

Monday 

•  Participant  survey  (14  of  cohort,  Team  #1) 

•  Participant  focus  group  x2  (Team  #1  +  #2) 

•  Participant  survey  (14  of  cohort,  Team  #1 ) 

•  Participant  focus  group  x3  (Team  #1  +  #2) 

Tuesday 

•  Participant  focus  group  x3  (Team  #1  +  #2) 

•  Leader  survey/focus  group  xl  (Team  #2) 

•  Participant  focus  group  x4  (Team#1  +#2) 

Wednesday 

•  Leader  survey/focus  group  x4  (Team  #1  +  #2) 

•  Leader  survey/focus  group  x4  (Team  #1  +  #2) 

Thursday 

•  Leader  survey/focus  group  x3  (Team  #1  +  #2) 

•  Peer  survey/focus  group  xl  (Team  #2) 

•  Peer  survey/focus  group  x4  (Team  #1  +#2) 

Friday 

•  Peer  survey/focus  group  x4  (Team  #1  +#2) 

•  Peer  survey/focus  group  x3  (Team  #1  +#2) 

•  Unit  records  data  collection  (Team  #2) 

8.  Data  Management  Procedures 

a.  General.  The  data  manager  will  receive  and  store  all  original  data  (questionnaires,  SJT  forms,  etc.) 
submitted  by  the  data  collection  teams.  S/he  will  maintain  all  digital  data  in  a  central  computerized 
database. 

b.  Source  Identification.  Each  Soldier  who  contributes  data  ( participants  and  associates)  will  enter  a 
unique  identification  number  (ID#)  on  all  data  collection  instruments.  The  ID#  will  consist  of  the  last 
letter  of  the  Soldier’s  last  name  plus  the  6  digits  (YYMMDD)  of  a  parent’s  birth  date  (e.g.,  T1 10913). 
The  ID#s  will  serve  as  prime  tags  for  all  Soldier  data.  Researchers  who  contribute  data 
(observations,  lessons  learned,  etc.)  may  also  be  assigned  ID#s. 

c.  Master  Data  Inventory.  A  master  list  of  data  received,  stored,  and  entered  into  the  database  will  be 
maintained  by  the  data  manager  in  spreadsheet  form.  Missing  data  will  be  annotated  and  tracked 
back  to  the  data  collector  for  possible  recovery.  As  drop-outs  are  identified,  their  records  will  be 
flagged. 

d.  Database  Entry.  The  process  for  entering  hardcopy  and  digital  data  into  the  database  will  be 
specified  by  the  data  manager,  who  will  carefully  monitor  the  process  for  completeness  and 
accuracy.  The  master  inventory  of  data  will  account  for  all  items  entered  into  the  database. 

e.  Quality  Control.  Data  entered  manually  into  the  database  will  be  inspected  (100%)  for  accuracy  and 
completeness  of  data  elements,  plus  completeness  of  users.  Any  data  entered  via  digital  transfer  or 
scanning  will  be  spot  checked  (-50%  sampling)  for  accuracy  and  completeness.  The  data  manager 
will  plan  and  supervise  the  quality  control  process. 

f.  Physical  Security.  As  a  general  rule,  every  Soldier  will  use  his/her  ID#  in  lieu  of  personal  name  on 
data  collection  instruments.  If  a  record  linking  personal  identifying  information  with  ID#s  is  kept,  the 
record  will  exist  under  strictly  controlled  conditions,  accessible  to  the  data  manager  and  evaluation 
manager  only.  Hardcopy  data  furnished  by  Soldiers  (e.g.,  questionnaires,  SJT  forms)  will  be  stored 
under  controlled  access  conditions  and  will  not  be  disseminated  outside  the  research  team.  Only 
research  investigators  will  have  access  to  the  database. 
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g.  Procedural  Changes.  All  deliberate  changes  in  the  standard  data  collection  and  processing 
procedures  must  be  approved  by  the  evaluation  manager.  A  cumulative  record  of  all  approved 
changes  will  be  maintained  by  the  data  manager.  Changes  that  affect  data  collectors  will  be 
disseminated  in  written  form.  The  Data  Collector’s  Guide  may  be  updated  as  necessary. 

h.  Data  Archiving.  All  raw  data  (collected  via  hardcopy  and  voice  recording  means)  and  database 
contents  will  be  archived  at  the  end  of  the  evaluation.  The  data  will  be  retained  or  disposed  of  in 
accordance  with  Army  and  ARI  policies. 

9.  Database  Requirements.  To  ensure  comprehensive  and  reliable  management  of  data,  the  central 
database  must  provide  specific  capabilities — including  relationality,  flexible  querying,  privacy,  and 
exportability  of  data  (especially  compatibility  with  SPSS®). 

a.  Responsibilities.  The  data  manager  and  evaluation  manager  will  specify  the  functional  requirements 
and  organization  of  the  database,  as  well  as  access  privileges.  The  data  manager  and  database 
technician  will  design  and  develop  the  database,  manage/execute  database  operations,  and  control 
access  to  the  database. 

b.  Basic  Functionality.  The  basic  capabilities  must  span  (1)  handling  of  multiple  data  types  including 
text,  (2)  structuring  in  accordance  with  data  sources,  (3)  open-ended  accommodation  of  records,  (4) 
manual  and  digital  data  entry,  (5)  quality  control  checking,  (6)  manipulation  and  compilation  of  data, 
and  (7)  data  security. 

c.  Data  Organization.  The  data  manager  will  establish  and  control  the  organization  of  the  database,  in 
collaboration  with  the  evaluation  manager.  In  general,  the  database  organization  will  mirror  the 
structure  of  the  data  collection  instruments.  The  structure  must  facilitate  the  aggregation  and 
analysis  of  data.  The  data  manager  will  create  and  maintain  a  master  schematic  of  the  database 
structure.  Once  the  structure  is  finalized  (following  pilot  testing),  no  changes  will  be  made  without 
the  evaluation  manager’s  approval. 

d.  Relationality.  It  is  essential  to  enable  linkage  of  data  across  records  for  a  given  participant.  Every 
record  or  separable  component  must  carry  the  ID#  for  the  unique  participant.  In  addition,  the 
database  must  support  aggregation  of  data  for  course  mode,  unit,  and  installation.  See  paragraph 
4d  above  for  critical  data  tags. 

e.  Database  Privacy.  No  personal  identifying  information  (e.g.,  SSN,  surname)  will  be  stored  in  the 
database.  A  list  of  persons  authorized  to  access  the  database  will  be  created  by  the  data  manager, 
approved  by  the  evaluation  manager,  and  updated  at  least  monthly. 

f.  Query  Options.  Research  team  members  (evaluation  manager,  data  manager,  investigators)  will 
use  direct  query  capabilities  to  support  data  analysis. 

g.  Exportability.  The  database  will  be  capable  of  outputting  data  sets  for  analysis  using  other  software 
(e.g.,  SPSS®,  Microsoft  Excel®). 

h.  Documentation.  The  data  manager  and  database  technician  will  informally  document  the  structure, 
functionality,  user  interface,  quality  control,  and  privacy  aspects  of  the  database.  The 
documentation  will  be  updated  if  substantial  changes  occur. 

10.  Resources 

a.  Staffing  for  Data  Collection  and  Management. 

(1)  Data  Collection  Workload. 

(a)  Course  Snapshot  Phase.  The  number  of  data  collection  sessions  in  a  course  snapshot 
week  would  generally  not  exceed  7  for  schoolhouse  and  MTT  courses.  Because  of  the 
group  size  or  nature  of  a  session,  each  would  be  handled  by  a  data  collection  team. 
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(b)  End  State  Assessment  Phase.  For  a  schoolhouse  cohort,  end  state  data  collection  would 
take  place  at  the  various  installations  (up  to  6)  where  the  participants  work.  If  the 
schoolhouse  participants  are  distributed  more  or  less  evenly  across  6  installations,  the 
number  of  data  collection  sessions  per  installation  (typically  less  than  20;  see  Table  2) 
could  be  handled  by  one  data  collection  team.  For  an  MTT  cohort,  however,  the  number  of 
sessions  (nearly  40,  as  seen  in  Table  4,  with  all  participants  located  at  the  same 
installation)  would  require  two  data  collection  teams  to  gather  all  data  within  a  week.  Most 
end  state  events  would  engage  two  data  collectors  per  session. 

(2)  Distribution  of  Workload.  Table  5  combines  the  workload  projections  and  the  scheduling 
factors  to  portray  a  notional  week-by-week  layout  of  data  collection  sessions,  by  cohort.  The 
workload  distribution  shown  in  Table  5  represents  a  best-case  scenario  where  course 
schedules  and  unit  schedules  would  allow  optimal  timing  of  the  data  collection  events.  During 
the  course  snapshot  phase  the  schedule  of  courses  would  involve  no  overlap  of  start  dates. 
During  the  end  state  assessment  phase,  the  schedule  would  be  orchestrated  so  that  no  more 
than  2  installations  involving  schoolhouse  sub-samples  (or  1  installation  involving  MTT  sub¬ 
samples)  fall  in  the  same  week.  The  best-case  scenario  rests  on  several  critical  assumptions 
(see  10a(6)  below),  at  least  some  of  which  are  unlikely  to  prove  valid.  The  actual  data 
collection  schedule  will  probably  involve  less-than-ideal  spacing  of  events  within  and  across 
installations,  creating  surge  periods  and  stretching  out  the  overall  data  collection  window. 

(3)  Data  Collection  Teams.  A  data  collection  team  will  consist  of  one  data  collector/SME  (to  serve 
as  a  facilitator)  and  one  data  collector/general.  During  the  end  state  phase,  the  majority  of  the 
sessions  will  require  both  team  members  working  together.  As  Table  5  shows,  a  best-case 
distribution  of  end  state  sessions  would  require  no  more  than  two  data  collection  teams  at  one 
time.  However,  three  (or  even  four)  data  collection  teams  would  provide  flexibility  to 
accommodate  surge  periods  driven  by  unit  schedule  constraints.  The  higher  level  of  staffing 
would  also  provide  backup  in  case  of  illnesses,  Family  emergencies,  etc.  Such  flexibility  may 
be  desirable  to  minimize  lost  or  degraded  data. 

(4)  Weekend  Scheduling.  Course  schedules  or  unit  constraints  may  lead  to  scheduling  data 
collection  events  on  weekends,  especially  during  the  course  snapshot  phase.  This  would  make 
it  necessary  for  data  collection  teams  to  work  during  occasional  weekends. 

(5)  Positions  Required.  The  following  positions  are  required  to  manage  and  conduct  data 
collection  and  database  activities,  with  backup  options  for  data  collectors: 

•  Data  collectors/general  (1-2  during  course  snapshots,  2-4  during  end  state  phase) 

•  Data  collectors/SMEs  (1-2  for  course-end  focus  groups,  2-4  during  end  state  phase) 

•  Evaluation  manager  (1) 

•  Data  manager,  with  database  experience  (1) 

•  Database  technician  (1) 

•  Data  entry  clerk  (1) 

(6)  Assumptions.  The  best-case  schedule  depicted  in  Table  5  depends  on  several  critical 
assumptions.  If  one  or  more  assumptions  prove  false,  the  evaluation  resource  plan  should 
provide  reasonable  flexibility  to  handle  contingencies.  The  critical  assumptions  are: 

•  Two  schoolhouse  classes  and  three  MTT  classes  will  be  scheduled  within  10-12  weeks, 
enabling  a  compact  distribution  of  data  collection  sessions. 

•  No  MTT  course  will  start  at  the  same  time  as  a  schoolhouse  course  or  another  MTT  course, 
minimizing  overlap  of  course  snapshot  activities. 

•  For  a  schoolhouse  cohort  during  the  end  state  phase,  data  collection  will  be  required  at  no 
more  than  two  installations  at  a  time. 

•  For  an  MTT  cohort  during  the  end  state  phase,  all  participants  and  associates  will  be 
located  at  a  single  installation. 
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Table  5.  Best-Case  Distribution  of  Data  Collection  Sessions  (All  Cohorts) 

«  Number  of  specific  sessions  (and  installations)  by  week  » 


Schoolhouse  Cohorts  (2  each) 

MTT  Cohorts  (3  each) 

Course  Snapshot  Phase 

Survey  #1 

T 

tfc 

h 

3 

8 

i 

$ 

8 

h 

5 

Focus  Gp 

Survey  #1 

T 

tfc 

t 

3 

Survey  #2 

8 

t 

5 

Focus  Gp 

1 

1(1) 

1(1) 

2 

1(1) 

1(1) 

3 

4 

i(i) 

i(i) 

5(1) 

1(1) 

1(1) 

5 

1(1) 

i(i) 

5(1) 

6 

7 

1(1) 

i(i) 

5(1) 

8 

1(1) 

1(1) 

9 

1(1) 

1(1) 

10 

11 

i(i) 

i(i) 

5(1) 

12 

1(1) 

i(i) 

5(1) 

Application/Incubation  Phase 

End  State  Assessment  Phase 

Parrtidpant 

Survey 

Participant 

Focus  Gp 

Unit  Leader 

Surv+Focus  Gp 

Unit  Peer 

Surv+Focus  Gp 

Records-Based 

Capture 

Parrtidpant 

Survey 

Parti  a  pant 

Focus  Gp 

Unit  Leader 

Surv+Focus  Gp 

Unit  Peer 

Surv+Focus  Gp 

Records-Based 

Capture 

24 

4(2) 

8(2) 

10(2) 

8(2) 

8(2) 

25 

4(2) 

8(2) 

10(2) 

8(2) 

8(2) 

26 

4(2) 

8(2) 

10(2) 

8(2) 

8(2) 

27 

28 

2(1) 

12(1) 

12(1) 

12(1) 

29 

6(1) 

30 

2(1) 

12(1) 

12(1) 

12(1) 

31 

6(1) 

32 

33 

2(1) 

12(1) 

12(1) 

12(1) 

34 

6(1) 

35 

4(2) 

8(2) 

10(2) 

8(2) 

8(2) 

36 

4(2) 

8(2) 

10(2) 

8(2) 

8(2) 

37 

4(2) 

8(2) 

10(2) 

8(2) 

8(2) 

38 

Notes:  (a)  The  first  entry  in  a  cell  gives  the  number  of  event-specific  sessions  for  the  week.  The  second  entry  (in 
parentheses)  is  the  number  of  installations,  (b)  Entries  in  bold  indicate  2  data  collectors  per  session,  (c)  If  timing 
dictates,  records-based  capture  sessions  will  be  scheduled  Monday  of  the  week  following  primary  data  collection. 
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•  During  the  end  state  phase,  the  number  of  sessions  at  a  given  installation  will  not  exceed 
what  two  data  collection  teams  can  handle  in  a  week  (~40  discrete  sessions). 

•  Units  at  a  given  installation  will  schedule  troops  for  end  state  data  collection  activities  within 
a  one-week  period  that  fits  the  evaluation  schedule. 

•  Units  will  support  by-name  tasking  for  specific  groups  with  no  more  than  5%  failure, 
eliminating  the  need  for  alternate  or  make-up  sessions. 

•  Backup  data  collector(s)  would  be  available  in  case  illnesses,  Family  emergencies,  or  other 
contingencies  occur. 

b.  Facilities. 

(1)  Data  Collection  Facilities.  When  collecting  data  in  the  schoolhouse,  data  collectors  will  use 
classroom  space  provided  by  the  schoolhouse.  At  FORSCOM  installations  they  will  use 
Government-furnished  classrooms,  conference  rooms,  or  office  space. 

(2)  On-site  Office  Space.  When  working  at  a  Government  site,  data  collectors  and  other 
researchers  will  use  temporary  office  space  provided  by  the  unit  or  installation.  Office 
furnishings  and  telephone  service  in  these  offices  will  be  provided  by  the  Government. 

c.  Equipment,  Software,  and  Supplies. 

(1)  Computers  and  Networks.  Each  data  collection  team  will  require  a  laptop  computer  for 
recording  notes  during  focus  groups.  Internet  access  by  data  collectors  while  on  site  is  not 
expected  to  depend  on  Government-furnished  networks. 

(2)  Database  Flost.  A  Government-furnished  computer  will  be  used  to  host  the  database,  with  no 
requirement  for  server-client  capabilities. 

(3)  Software.  Special  software  will  be  required  to  implement  machine  scanning  of  survey 
questionnaires  and  perhaps  database  capabilities. 

(4)  Projection  Equipment.  If  projection  of  slides  is  required  (e.g.,  to  display  focus  group  materials), 
the  data  collectors  should  be  prepared  to  provide  suitable  projection  equipment.  Providing 
materials  in  handout  form  is  encouraged,  rather  than  projecting  them. 

(5)  Supplies.  Data  collectors  should  take  pencils  and/or  pens  to  survey  sessions.  If  they  are 
required  to  take  bulky  materials  such  as  note  tablets  and  butcher  paper  to  focus  group 
sessions,  such  supplies  would  likely  be  purchased  locally. 

11.  Data  Collection  Locations 

a.  Schoolhouse.  The  resident  Infantry  ALC  Phase  II  course  is  taught  at  Fort  Benning,  GA.  The 
collection  of  data  from  students  attending  the  course  will  occur  at  Fort  Benning.  No  other  locations 
for  schoolhouse  data  collection  are  anticipated. 

b.  FORSCOM  Units.  The  sampling  plan  targets  MTT  courses  that  will  be  taught  at  three  different 
installations  in  CONUS.  The  schoolhouse  participants  will  return  to  infantry  units  that  could  be 
located  at  eight  different  CONUS  installations.  To  reduce  evaluation  costs,  this  plan  assumes  that 
data  collection  will  involve  no  more  than  6  installations,  as  follows: 

♦  Fort  Benning  ♦  Fort  Hood  ♦  Fort  Riley 

♦  Fort  Lewis  ♦  Fort  Drum  ♦  Fort  Stewart 

12.  Pilot  Test 

a.  Purpose.  A  pilot  test  will  be  essential  to  ensure  the  quality  and  readiness  of  the  data  collection  and 
management  procedures  and  tools.  It  will  also  test  the  infrastructure  supporting  the  data  collection 
activities. 
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b.  Concept.  The  pilot  test  will  be  a  trial  implementation  with  all  players  (Soldiers  and  researchers) 
represented  credibly.  A  compressed  schedule  for  an  ALC  cohort  will  be  followed,  with  data 
collection  events  simulated  in  mock-up  fashion.  All  data  collection  instruments  will  be  used.  The 
complete  process  for  collecting  and  managing  data  (including  database  operations)  will  be 
exercised.  The  results  of  the  test  will  be  used  to  revise  data  collection/management  tools  and 
procedures,  as  necessary. 

c.  Pilot  Test  Plan.  The  outline  plan  for  the  pilot  test  appears  in  Appendix  G  (Pilot  Test  -  Outline  Plan). 
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NOTIONAL  SCHEDULE  FOR  MTT  COHORT 

(Based  on  1-week  end-state  window  with  2  data  collection  teams  working  at  1  installation) 


Date 

Time 

Event 

Comments 

Course  Snapshot  Phase 

12  Jan  10 

1000-1100 

Student  Survey  (start  of  course) 

Entire  class  ( n  =  -100) 

1100-1200 

Student  SJT  (start  of  course) 

Entire  class  ( n  =  -100) 

4  Feb  10 

0800-0830 

Student  Survey  (end  of  course) 

Entire  class  ( n  =  -100) 

0830-0930 

Student  SJT  (end  of  course) 

Entire  class  ( n  =  -100) 

0930-1130 

Student  Focus  Group  (Group  #1) 

Subsample  ( n  =  5) 

1300-1500 

Student  Focus  Group  (Group  #2) 

Subsample  ( n  =  6) 

1500-1700 

Student  Focus  Group  (Group  #3) 

Subsample  ( n  =  5) 

5  Feb  10 

0800-1000 

Student  Focus  Group  (Group  #4) 

Subsample  (n  =  6) 

1000-1200 

Student  Focus  Group  (Group  #5) 

Subsample  (n  =  5) 

End  State  Assessment  Phase 

26  Jul 10 

0900-1000 

Participant  Survey  (14  of  sample) 

DC  Team  #1 

1000-1200 

Participant  Focus  Group  (Groups  #P1  &  #P2) 

DC  Teams  #1  and  #2 

1300-1400 

Participant  Survey  (14  of  sample) 

DC  Team  #1 

1300-1500 

Participant  Focus  Group  (Group  #P3) 

DC  Team  #2 

1500-1700 

Participant  Focus  Group  (Groups  #P4  &  #P5) 

DC  Teams  #1  and  #2 

27  Jul 10 

0800-1000 

Participant  Focus  Group  (Groups  #P6  &  #P7) 

DC  Teams  #1  and  #2 

1000-1200 

Participant  Focus  Group  (Group  #P8) 

DC  Team  #1 

Leader  Survey  +  Focus  Group  (Group  #L1) 

DC  Team  #2 

1300-1500 

Participant  Focus  Group  (Groups  #P9  &  #P10) 

DC  Teams  #1  and  #2 

1500-1700 

Participant  Focus  Group  (Groups  #P  1 1  &  #P12) 

DC  Teams  #1  and  #2 

28  Jul 10 

0800-1000 

Leader  Survey  +  Focus  Group  (Group  s#L2  &  #L3) 

DC  Teams  #1  and  #2 

1000-1200 

Leader  Survey  +  Focus  Group  (Groups  #L4  &  #L5) 

DC  Teams  #1  and  #2 

1300-1500 

Leader  Survey  +  Focus  Group  (Groups  #L6  &  #L7) 

DC  Teams  #1  and  #2 

1500-1700 

Leader  Survey  +  Focus  Group  (Groups  #L8  &  #L9) 

DC  Teams  #1  and  #2 

29  Jul 10 

0800-1000 

Leader  Survey  +  Focus  Group  (Groups  #L10  &  #L1 1) 

DC  Teams  #1  and  #2 

1000-1200 

Leader  Survey  +  Focus  Group  (Group  #L12) 

DC  Team  #1 

Peer  Survey  +  Focus  Group  (Group  #Pr1) 

DC  Team  #2 

1300-1500 

Peer  Survey  +  Focus  Group  (Groups  #Pr2  &  #Pr3) 

DC  Teams  #1  and  #2 

1500-1700 

Peer  Survey  +  Focus  Group  (Groups  #Pr4  &  #Pr5) 

DC  Teams  #1  and  #2 

30  Jul 10 

0800-1000 

Peer  Survey  +  Focus  Group  (Groups  #Pr6  &  #Pr7) 

DC  Teams  #1  and  #2 

1000-1200 

Peer  Survey  +  Focus  Group  (Groups  #Pr8  &  #Pr9) 

DC  Teams  #1  and  #2 

1300-1500 

Peer  Survey  +  Focus  Group  (Groups  #Pr1 0  &  #Pr1 1 ) 

DC  Teams  #1  and  #2 

1500-1700 

Peer  Survey  +  Focus  Group  (Group  #Pr12) 

DC  Team  #1 

Unit  Records  Data  Collection 

DC  Team  #2 

NOTE-1:  Focus  group  size  =  4-6  Soldiers  ( participants  and  peers)  or  3-5  Soldiers  (leaders). 

NOTE-2:  All  rostered  leaders  and  peers  will  be  scheduled  for  surveys  to  ensure  100%  sampling,  with  excess 
individuals  excused  from  the  focus  group  sessions  (60-75%  sampling)  as  necessary. 
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PILOT  TEST  -  OUTLINE  PLAN 


Test  Aspect 

Plan 

Objective 

Verify  that  data  collection  instruments,  research  personnel,  infrastructure,  and  data 
collection/management  procedures  are  ready  for  executing  the  FarXfer  evaluation. 

Location 

Fort  Benning,  GA  and  one  remote  site 

Dates 

2-3  days  during  the  week  of  14  Dec  09  (or  2-4  weeks  before  start  of  data  collection) 

Staffing 

•  Participants  -  NCOs  in  casual  status  at  Fort  Benning  (3-4  ea) 

•  Associates  (Soldiers  in  casual  status)  -  leader  (1LT  x2),  peer  (2  ea) 

•  Research  team  -  ARI  investigator(s)  and  contract  personnel,  including  Evaluation  Mgr 

Test  Materials 

•  Data  collection  instruments  (DCIs) 

•  Data  Collection  and  Management  Plan 

•  Data  Collector’s  Guide 

Test  Environment 

•  Data  collection  facilities  =  ARI  workspace,  contractor  workspace,  player  offices 

•  Equipment  =  Govt  database  host,  project  laptops  (3  ea),  PCs  normally  used  by  players 

•  Connectivity  via  Internet 

Readiness  Criteria 

•  Mature  test  materials  ready  for  implementation 

•  All  Soldiers  and  researchers  on  hand,  including  2  fully  trained  data  collectors 

•  Database  attended  by  database  technician 

•  Go-ahead  decision  by  the  evaluation  manager 

Test  Management 

•  ARI  coordinates  for  Soldiers  to  serve  as  participants  and  associates 

•  Detailed  test  schedule  guides  sequence  and  pacing  of  pilot  test  activities 

•  Evaluation  manager  oversees  test  activities  and  resolves  test  issues 

•  Data  manager  supervises  data  collection  and  management  activities 

•  End-of-day  hot  washes  facilitate  progress  tracking  and  mid-course  adjustments 

•  Evaluation  manager  decides  when  to  end  pilot  testing,  based  on  sufficiency  of  data 

Test  Procedures 

•  Players  execute  events  IAW  Data  Collection  and  Management  Plan 

•  Facilitator  conducts  and  documents  focus  group  sessions  (2  iterations) 

•  Each  participant  completes  and  submits  checklist  in  dry-run  fashion 

•  Data  collectors  gather  and  submit  survey,  SJT,  focus  group,  &  other  data 

•  Data  collectors  gather  and  submit  records-based  data  in  dry-run  fashion 

•  Data  manager  maintains  checklist  as  events/milestones  are  completed 

•  Data  manager  exercises  database  functions,  including  data  entry 

•  At  least  two  investigators  exercise  database  interface,  including  query  functions 

•  Players  retest  instruments  and  functions  as  modifications  are  completed 

•  Data  manager  determines  sufficiency  of  data  from  each  test  event 

Data  Collection 

•  Primary  means  =  DCIs  developed  for  the  evaluation  +  comment  sheets 

•  Secondary  means  =  facilitated  end-of-day  and  end-of-test  hotwashes 

•  Data  manager  collects  hardcopy  forms  and  tracks  all  data,  with  evaluator’s  help 

•  Evaluator  takes  notes  during  each  hotwash  and  prepares  summary 

•  Researchers  document  own  lessons  learned 

Data  Utilization 

•  Compile  and  organize  data  using  the  project  database 

•  Examine  raw  and  digital  data  for  flaws,  malfunctions,  problems,  and  discrepancies 

•  Identify  candidate  fixes  and  improvements  for  materials  and  procedures 

•  Resolve  issues,  problems,  and  improvements  during  pilot  test 

Privacy  Controls 

•  No  Soldier  identification  on  DCIs,  comment  sheets,  or  compilations 

•  Restricted  access  to  all  data 

•  Non-attribution  of  findings 

Outcomes 

•  Revised  evaluation  materials  and  procedures,  ready  for  execution  of  data  collection 

•  Lessons  learned  for  executing  the  FarXfer  evaluation 
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APPENDIX  H 


DATA  COLLECTION  INSTRUMENTS  -  OUTLINE  DESIGN 


Purpose:  This  document  outlines  the  design  of  the  data  collection  instruments  to  be  used  in 
measuring  the  impact  of  the  new  Infantry  ALC  POI  among  tactical  units. 

Privacy  Protection:  Personal  identification  numbers  will  be  assigned  to  all  Soldiers  and  used 
to  identify  individuals  on  all  Soldier-completed  forms. 


Instrument 

Design  Parameters 

Measurement  Scope 

1.  Survey  Questionnaires 

A.  Biographical 
Inventory 

•  Target  Audience:  ALC  students/graduates 

•  Mode:  Hardcopy  (self-explanatory,  stand-alone) 

•  Max  completion  time:  15  min 

•  Prompting:  maximum  use  of  itemized  lists,  with 
“other”  write-in  where  appropriate 

•  Modeling:  inclusion  of  example  entries  where 
guiding  is  desirable 

•  Layout:  boxed  items,  with  internal  borders  where 
multiple  elements  are  involved 

•  Demographic  characteristics  (rank,  TIS,  etc.) 

•  Schooling  -  military  and  civilian 

•  Assignment  history 

•  Deployment  history 

B-1.  ALC  Stu¬ 
dents  Survey, 
Start-of-Course 

•  Target  Audience:  ALC  students  (Day  1) 

•  Mode:  Hardcopy  (self-explanatory,  stand-alone) 

•  Max  completion  time:  30  min 

•  Primary  rating  scale:  uni-dimensional  level  of 
competence,  6-point,  no  neutral  point,  response 
by  circling  a  number  or  selecting  a  box 

•  Layout:  boxing  with  internal  borders,  embedded 
scale  codes,  ruled  write-in  boxes 

•  Clustering:  similar  items  presented  together  in 
multi-row  box,  with  shared  stem 

•  Write-ins:  elaboration  +  open-ended 

•  Extent  of  NCO  competencies  (per  metrics) 

•  Extent  of  NCO  competencies  (per  POI) 

•  Perceived  readiness  for  ALC  learning 

•  Motivation  to  learn  as  a  student 

•  Expected  job  relevance  of  ALC  training/learning 

•  Expected  ALC  outcomes 

B-2.  ALC  Stu¬ 
dents  Survey, 
End-of-Course 

•  Target  Audience:  ALC  students  (final  day) 

•  Mode:  Hardcopy  (self-explanatory,  stand-alone) 

•  Max  completion  time:  30  min 

•  Primary  rating  scale:  same  as  IB-1 

•  Layout:  same  as  IB-1 

•  Clustering:  same  as  IB-1 

•  Write-ins:  elaboration  +  open-ended 

•  Extent  of  ALC  training/practice/opportunities 

•  Extent  of  NCO  competencies  (per  metrics) 

•  Extent  of  NCO  competencies  (per  POI) 

•  Selected  ALC  learning  outcomes 

•  Expected  job  relevance  of  ALC  training/learning 

•  Realized  ALC  outcomes 

C.  Participants 
Survey,  End  State 

•  Target  Audience:  ALC  graduates 

•  Mode:  Hardcopy  (self-explanatory,  stand-alone) 

•  Max  completion  time:  45  min 

•  Rating  scale:  bipolar  agree-disagree,  6-point,  no 
neutral  point,  “not  performed”  option,  response 
by  circling  a  number  or  selecting  a  box 

•  Layout:  boxing  with  internal  borders,  embedded 
scale  codes,  ruled  write-in  boxes 

•  Clustering:  similar  items  presented  together  in 
multi-row  box,  with  shared  stem 

•  Write-ins:  elaboration  +  open-ended 

•  Sufficiency  of  application  opportunities 

•  Job  relevance  of  ALC  training/learning 

•  Perceived  learning  transfer 

•  Change  in  KSAs/behavior/performance/attitudes 

•  Listing  of  selected  activities/decisions/choices 

•  Change  in  unit  aspects  (e.g.,  cohesion) 

•  Positive  feedback  received  from  others 

•  Contributions  to  unit  operations  and  processes 

•  Change  in  standing/stature  within  the  unit 

•  Benefits  to  individual  and  unit 

•  Unit  environment  as  a  contributing  factor 
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Instrument 

Design  Parameters 

Measurement  Scope 

D.  Unit  Leaders 
Survey 

•  Target  Audience:  Leaders  with  direct  knowledge 
of  named  NCO  (Pit  Sgt,  Pit  Ldr,  Co  1SG,  Co  Cdr) 

•  Mode:  Hardcopy  (self-explanatory,  stand-alone) 

•  Max  completion  time:  30  min 

•  Rating  scale:  same  as  1C 

•  Layout:  same  as  1C 

•  Clustering:  same  as  1C 

•  Write-ins:  elaboration  +  open-ended 

•  Opportunities  for  NCO  to  apply  ALC  learning 

•  Change  in  KSAs/behavior/performance/attitudes 

•  Change  in  unit  aspects  (e.g.,  cohesion) 

•  Contributions  to  unit  operations  and  processes 

•  Change  in  standing/stature  within  the  unit 

•  Benefits  to  individual  and  unit 

•  Listing  of  new  equipment  and  TTP 

E.  Platoon 
Sergeants  Survey 

•  Target  Audience:  Pit  Sgts  to  whom  ALC 
graduates  report  (subset  of  unit  leaders) 

•  Mode:  Hardcopy  (self-explanatory,  stand-alone) 

•  Max  completion  time:  10  min 

•  Placement:  special  section  of  Unit  Leaders 
Survey(\D) 

•  Rating  scale:  same  as  1C 

•  Layout:  same  as  1C 

•  Clustering:  same  as  1C 

•  Write-ins:  elaboration  +  open-ended 

•  Change  in  self-development  behavior/attitudes 

•  Listing  of  selected  self-development  activities 

•  Recognition  of  NCO  for  applying  ALC  learning 

F.  Unit  Peers 
Survey 

•  Target  Audience:  other  squad  leaders  within 
NCO’s  company 

•  Mode:  Hardcopy  (self-explanatory,  stand-alone) 

•  Max  completion  time:  30  min 

•  Placement:  may  be  combined  with  Unit  Leaders 
Survey(\D) 

•  Rating  scale:  same  as  1C 

•  Layout:  same  as  1C 

•  Clustering:  same  as  1C 

•  Write-ins:  elaboration  +  open-ended 

•  Change  in  KSAs/behavior/performance/attitudes 

•  Change  in  unit  aspects  (e.g.,  cohesion) 

•  Contributions  to  unit  operations  and  processes 

•  Change  in  standing/stature  within  the  unit 

•  Benefits  to  unit 

II.  Focus  Group  Protocols 

A.  ALC  Students 
Focus  Group 
(End-of-Course) 

•  Target  Audience:  ALC  graduates 

•  Mode:  Group  containing  5-6  participants  (pure) 

•  Max  session  time:  120  min 

•  Investigator  team:  facilitator  (SME)  +  note  taker 

•  Max  #  questions:  25  (50%  hit  rate) 

•  Structure:  organized  by  category,  ordered  by 
priority  within  category 

•  Activities:  discussion,  problem  solving,  what-if 
brainstorming 

•  Instructions:  geared  for  diverse  mix  of  facilitators 

•  Extent  of  ALC  training/practice/opportunities 

•  Selected  ALC  learning  outcomes 

•  Expected  application  environment 

•  Expected  relevance  of  ALC  training/learning 

•  ALC  training  issues 

•  ALC  program  improvement 

B.  Participants 
Focus  Group 
(End  State) 

•  Target  Audience:  Participants 

•  Mode:  Group  containing  4-6  participants  (pure) 

•  Max  session  time:  120  min 

•  Investigator  team:  facilitator  (SME)  +  note  taker 

•  Max  #  questions:  25  (50%  hit  rate) 

•  Structure:  organized  by  category,  ordered  by 
priority  within  category 

•  Activities:  discussion,  problem  solving,  what-if 
brainstorming 

•  Instructions:  geared  for  diverse  mix  of  facilitators 

•  Follow-up  of  survey  results 

•  Opportunities  for  selected  KSA  application 

•  Change  in  KSAs/behavior/performance/attitudes 

•  Listing  of  selected  activities 

•  Examples  of  behaviors  and  ALC  impact 

•  Contributions  to  unit  operations  and  processes 

•  Application  environment  and  enablers 

•  Benefits  to  individual  and  unit 

•  ALC  program  improvement 
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Instrument 

Design  Parameters 

Measurement  Scope 

C.  Unit  Leaders 
Focus  Group 

•  Target  Audience:  Leaders  with  direct  knowledge 
of  named  NCO  (Pit  Sgt,  Pit  Ldr,  Co  1SG,  Co  Cdr) 

•  Mode:  Group  containing  3-5  leaders  (pure) 

•  Max  session  time:  90  min 

•  Investigator  team:  facilitator  (SME)  +  note  taker 

•  Max  #  questions:  20  (50%  hit  rate) 

•  Structure:  organized  by  category,  ordered  by 
priority  within  category 

•  Activities:  discussion,  problem  solving,  what-if 
brainstorming 

•  Instructions:  geared  for  diverse  mix  of  facilitators 

•  Opportunities  for  selected  KSA  application 

•  Change  in  KSAs/behavior/performance/attitudes 

•  Listing  of  selected  activities  [PSG  only] 

•  Examples  of  behaviors  and  ALC  impact 

•  Contributions  to  unit  operations  and  processes 

•  Benefits  to  individual  and  unit 

•  ALC  program  improvement 

D.  Unit  Peers 

Focus  Group 

•  Target  Audience:  other  squad  leaders  within 
NCO’s  company 

•  Mode:  Group  containing  4-6  peers  (pure) 

•  Max  session  time:  90  min 

•  Investigator  team:  facilitator  (SME)  +  note  taker 

•  Max  #  questions:  20  (60%  hit  rate) 

•  Structure:  organized  by  category,  ordered  by 
priority  within  category 

•  Activities:  discussion,  problem  solving,  what-if 
brainstorming 

•  Instructions:  geared  for  diverse  mix  of  facilitators 

•  Placement:  protocol  may  be  subset  of  Unit 
Leaders  Focus  Gtoup(IIC) 

•  Change  in  KSAs/behavior/performance/attitudes 

•  Examples  of  behaviors  and  ALC  impact 

•  Contributions  to  unit  operations  and  processes 

•  Application  environment  and  enablers 

•  Benefits  to  individual  and  unit 

•  ALC  program  improvement 

III.  Miscellaneous  Instruments 

A.  Situational 
Judgment  Tests 
(Alternate  Ver¬ 
sions) 

•  Target  Audience:  ALC  students 

•  Mode:  Hardcopy  (self-guided,  stand-alone) 

•  Max  session  time:  45  min 

•  Max  #  questions:  35  (per  session) 

•  Question  structure:  tag  line  +  mini-scenario  + 
response  options 

•  Response  type:  multiple  choice 

•  Internal  linkage:  none  (independent  questions) 

•  Investigator  team:  monitors  (two  per  classroom) 

•  Instructions:  geared  for  target  audience 

•  Versions:  2  each,  psychometrically  equivalent 

•  Test  conditions:  closed  book,  no  collaboration, 
enforced  time  limit 

•  Scoring:  comparison  with  expert  judgment 

•  Critical  NCO  competencies  (per  FarXfer  metrics) 

•  Critical  NCO  competencies  (per  ALC  POI) 

B.  Application 
Checklist  (Recur¬ 
ring) 

•  Users:  ALC  graduates  working  in  units 

•  Mode:  Hardcopy  (self-guided,  stand-alone) 

•  Distribution:  through  unit  action  officer 

•  Iteration:  bi-weekly  recording  and  submission 

•  Prompting:  reminders  by  unit  action  officer 

•  Response  type:  check-the-box 

•  Structure:  organized  by  competency  category, 
with  columns  for  successive  weeks 

•  Layout:  boxing  with  internal  borders,  shading  of 
alternate  columns,  ruled  write-in  boxes 

•  Instructions:  geared  for  stand-alone  use  by 
junior  NCOs 

•  Frequency  of  using  ALC-acquired  KSAs 

•  Description  of  special-interest  activities 

•  Examples  of  KSA  application 

•  Explanatory  notes 
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Instrument 

Design  Parameters 

Measurement  Scope 

C.  Unit  Records- 
Based  Capture 

Form 

•  Users:  data  collectors  (researchers) 

•  Mode:  Hardcopy 

•  Source  mix:  personnel  reports,  UCMJ  admini¬ 
strative  records,  performance  test  records,  CTC 
take-home  packages 

•  Structure:  organized  by  section  (source), 
ordered  by  nature  of  data 

•  Instructions:  geared  for  diverse  data  collectors, 
tailored  by  section 

•  Adverse  personnel  status  incidents  (e.g.,  AWOL, 
early  reassignment) 

•  UCMJ  actions 

•  Individual  proficiency  scores  (APFT,  weapons 
qualification,  Common  Tasks,  Warrior  Tasks) 

•  Collective  proficiency  evaluations  (Battle  Drills, 
squad/  section/crew  tasks) 

D.  Master  Data 
Inventory 

•  Users:  Data  Manager 

•  Mode:  digital  (spreadsheet) 

•  Structure:  open-ended,  organized  by  instrument 
and  source  ID,  ordered  chronologically 

•  Tagging:  standard  filename  conventions  +  code 
for  location  in  database 

•  Index:  by  participant,  cohort,  unit,  instrument 

•  Instructions:  geared  for  investigators 

•  Monitoring:  flags  visible  for  pending/missing  data 
and  pending  actions 

•  Identification  (instrument,  date,  unit,  participants, 
location,  researchers,  etc.) 

•  Location  of  raw  data 

•  Status  of  data  entry,  including  filename(s)  or 
spreadsheet  name(s) 

•  Status  of  QA  review 

•  Explanatory  notes  (e.g.,  procedural  departures, 
drop-outs) 

Underlying  Principles: 

•  Government  computers  are  unlikely  to  be  available  in  participating  platoons  and  squads, 
limiting  the  potential  for  online  administration  of  questionnaires 

•  To  avoid  impeding  focus  group  dynamics,  mixed  groups  (e.g.,  leaders  and  peers 
together),  should  be  avoided  in  favor  of  “pure”  (unmixed)  mode 

•  A  focus  group  can  address  pointed  questions  in  problem-solving  mode,  including  “what  if” 
challenges  and  collaborative  brainstorming  options 

•  Focus  group  discussion  of  one  topic  may  take  5-15  min,  depending  on  the  complexity  of 
the  challenge 

•  While  a  focus  group  can  include  simple  interview-like  discussions,  emphasizing  group 
brainstorming  or  problem  solving  can  produce  more  valuable  insights. 

•  A  focus  group  protocol  would  contain  a  pool  of  questions,  from  which  a  facilitator  would 
choose  a  manageable  subset  for  a  given  session 

•  A  focus  group  facilitator  should  have  military  experience  commensurate  with  the  Soldiers, 
but  a  note  taker  could  have  less  military  experience 

•  The  note  taker  would  voice-record  a  focus  group  session  to  back-up  his/her  notes,  but 
preparing  transcripts  from  voice  recordings  is  unnecessary 

•  Performance  testing  (SJTs)  would  provide  evidence  of  actionable  knowledge  and  offer 
objective  measures  for  calculating  ALC-induced  learning  (pre-versus-post  deltas) 

•  To  determine  SJT  topics,  a  prioritization  process  can  guide  the  selection  of  critical  NCO 
competencies  from  the  FarXfer  metrics  and  ALC  POI 

•  Equivalent  versions  of  SJTs  can  be  constructed  to  avoid  carry-over  from  start-of-course  to 
end-of-course  sessions 

•  Questionnaires  and  SJTs  should  be  self-explanatory,  requiring  the  session  moderator  to 
provide  no  directions  other  than  administrative  information 

•  Researchers  will  be  allowed  to  copy  specific  data  elements  from  unit  records  without 
duplicating  actual  records 
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Special  Considerations: 

•  A  leader/peer  questionnaire  should  address  no  more  than  one  target  NCO-participant  at  a 
time 

•  If  a  leader  or  peer  provides  survey  information  for  more  than  one  NCO-participant, 
separate  sessions  should  be  scheduled 

•  The  size  of  a  focus  group  should  not  exceed  6  participants/peers  (or  5  leaders)  to  ensure 
full  participation  by  all 

•  For  focus  groups  involving  higher-ranking  leaders,  smaller  groups  (3-4  leaders)  may  be 
preferable 

•  The  size  of  a  focus  group  should  not  fall  below  3  Soldiers,  to  ensure  enough  individuals  for 
collaborative  brainstorming 

•  Discussion  of  a  specific  NCO’s  weaknesses  in  group  sessions  should  be  avoided  to 
maintain  privacy  protections 

•  Objective  measurement  of  specific  competencies  (via  SJTs)  could  influence  subjective 
ratings  of  the  same  competencies  (perhaps  a  positive  effect) 

•  Creating  time  pressures  during  SJT  execution  may  be  desirable  but  would  confound 
knowledge  recall/application  and  stress 

•  Pairing  the  SJT  instruments  with  the  start-of-course  and  end-of-course  questionnaires  may 
be  undesirable  because  it  could  compromise  the  student  survey  results 

•  Use  of  a  recurring  activity  checklist  by  participants  in  units  could  influence  their  patterns  of 
applying  ALC  skills  and  knowledge 
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APPENDIX  I 


EVALUATION  OF  INFANTRY  ALC  TRANSFER 


DATA  COLLECTOR’S  GUIDE 

for 

ALL  ACTIVITIES 


♦  ♦  A  Roadmap  for  High-Quality  Data  ♦  ♦ 


Point  of  Contact: 

Dr.  Robert  Pleban,  Robert.Pleban@us.army.mil,  (706)  545-2207 
ARI  -  Fort  Benning  Research  Unit 


March  2009 


DATA  COLLECTOR’S  GUIDE 


About  This  Guide 

This  Data  Collector's  Guide  is  the  primary  job  aid  for  ARI  team  members 
collecting  data  in  the  evaluation  of  Infantry  ALC  transfer.  Each  data 
collector  should  keep  it  handy  when  working  at  an  evaluation  site. 


1.  What’s  the  purpose  of  the  data  collection? 

♦  Build  a  comprehensive  picture  of  the  new  1 1 B  ALC  POI’s  effectiveness 

♦  Give  FORSCOM  units  a  voice  to  influence  Infantry  ALC  training 

♦  Generate  objective  information  for  TRADOC  decisions  on  the  ALC  program 

♦  Capture  good  ideas  for  improving  the  Infantry  ALC  POI 

2.  Who’s  on  the  data  collection  team? 

♦  Evaluation  Manager  -  Dr.  Robert  Pleban 

♦  Data  Manager  -  Dr.  Marisa  Miller 

♦  Data  Collectors/SMEs  -  [insert  names] 

♦  Data  Collectors/general  -  [insert  names] 

♦  Database  Technician  -  [insert  name] 

♦  Data  Entry  Clerk  -  [insert  name] 

3.  Where  are  the  data  being  collected? 

♦  Fort  Benning  ♦  Fort  Hood  ♦  Fort  Riley 

♦  Fort  Lewis  ♦  Fort  Drum  ♦  Fort  Stewart 

4.  What  are  the  questions  of  interest? 

♦  How  much  is  ALC  training  relevant  and  useful  to  NCOs  working  in  units? 

♦  How  does  ALC  impact  competencies  and  performance  of  squad  leaders? 

♦  How  does  ALC  impact  unit  processes,  performance,  and  operations? 

♦  What  benefits  does  ALC  bring  to  individual  NCOs  and  their  units? 

♦  What  unit  factors  and  events  influence  utilization  of  ALC  learning? 

5.  What  are  the  sources  of  data? 

♦  ALC  students/graduates  working  as  squad  leaders  -  participants 

♦  Leaders  and  peers  who  know  participants  -  associates 

♦  Training  records,  personnel  reports,  UCMJ  records,  etc.  -  unit  records 

♦  Data  collectors,  investigators,  data  manager  -  research  team 
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6.  What  are  the  phases  of  data  collection? 

♦  Start-up  -  training  and  rehearsal  of  data  collectors 

♦  Course  snapshots  -  of  students  at  start  and  end  of  ALC 

♦  Application/incubation  -  while  graduates  apply  what  they  learned  (5-6  mos  desired) 

♦  End  state  assessment  -  capture  of  data  on  ALC  impact 

♦  Close-out  -  orderly  wrap-up  following  final  data  collection 

7.  How  are  the  data  collected?  (See  Attachment  1 ) 

♦  Survey  questionnaires  ♦  Application  checklist 

♦  Focus  group  protocols  ♦  Records-based  capture  form 

♦  Situational  judgment  tests 

8.  What  are  the  main  duties  of  a  data  collector? 

♦  Schedule  course  snapshot  data  collection  sessions  -  see  Checklist  1 

♦  Schedule  end  state  data  collection  sessions  -  see  Checklist  2 

♦  Administer  survey  sessions  -  see  Checklist  3 

♦  Administer  situational  judgment  tests  -  see  Checklist  4 

♦  Conduct  focus  group  sessions  -  see  Checklist  5 

♦  Collect  data  from  unit  records  -  see  Checklist  6 

♦  Submit  data  to  the  Data  Manager  -  see  Checklist  7 

♦  Protect  and  store  data  locally  -  see  Checklist  8 

9.  What  are  the  eight  commandments  for  data  collectors? 

♦  Respect  the  busy  schedules  of  Soldiers  and  units  -  prepare  well 

♦  Safeguard  the  privacy  of  Soldiers  and  units  at  all  costs  -  protect  the  data 

♦  Honor  the  voluntary  participation  of  Soldiers  -  coerce  no  one 

♦  Follow  the  standard  procedures  -  produce  highest  quality  data 

♦  Don’t  change  procedures  -  unless  approved  by  the  Evaluation  Manager 

♦  Seek  advice  -  talk  with  the  Data  Manager  when  problems  or  issues  arise 

♦  Document  what  happens  -  especially  departures  from  the  standard  procedures 

♦  Complete  notes  and  annotations  quickly  -  memory  will  fail  too  soon 

10.  What  about  privacy  of  Soldiers  and  units? 

♦  Protecting  the  privacy  of  Soldiers  and  units  is  an  absolute  requirement 

♦  Every  Soldier  will  use  an  ID#  in  lieu  of  personal  name  or  SSN 

♦  ID#  =  last  letter  of  last  name  +  6-digit  birthday  of  a  parent  (YYMMDD) 

♦  Every  data  collector  must  safeguard  hardcopy  and  digital  data  in  his/her  custody 

♦  Specific  safeguard  procedures  are  spelled  out  in  this  guide’s  checklists 

11.  What  is  the  dress  code  for  data  collectors? 

♦  Business  casual  attire  (tie  or  jacket  optional)  will  be  worn  at  data  collection  sites 

♦  A  picture  identification  badge  will  be  worn  in  plain  sight  at  data  collection  sites 
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12.  How  does  travel  work  for  data  collectors? 

♦  All  travel  must  be  approved  by  the  Evaluation  Manager  or  Data  Manager 

♦  Site  visits  will  generally  last  1-2  weeks,  with  weekend  duty  as  required 

♦  A  designated  administrative  assistant  will  handle  all  travel  arrangements 

♦  Any  changes  to  travel  plans  must  be  made  thru  the  administrative  assistant 

♦  Travel-related  receipts  must  be  submitted  to  the  administrative  assistant 

13.  Where  to  get  additional  information? 

♦  Data  Collection  and  Management  Plan 

♦  FarXfer  Data  Collection  Instruments  -  Outline  Design 

14.  ARI’s  research  project ... 

♦  Helps  USAIS  and  NCO  Academy  keep  pace  with  Army  Transformation 

♦  Promotes  utilization  of  ALC  learning  among  operational  units 

♦  Contributes  to  the  Army’s  NCOES  modernization  efforts 

♦  Is  conducted  by  ARI’s  Fort  Benning  Research  Unit 

15.  Points  of  Contact 

♦  Dr.  Robert  Pleban,  Evaluation  Manager,  Robert.Pleban@us.army.mil,  (706)  545-2207 

♦  Dr.  Marisa  Miller,  Data  Manager,  Marisa.Miller@us.army.mil,  (706)  545-XXXX 

♦  XXXX,  Administrative  Asst,  xxxxxx 


1-4 


Attachment  1 


DATA  COLLECTION  ACTIVITIES 


Note-1\  Each  activity  has  a  corresponding  data  collection  instrument  that  must  be  used. 

Note-2\  During  the  application/incubation  period,  participants  will  complete  Application 
Checklists  every  pay  period.  Data  collectors  will  not  be  involved  in  this  activity. 


Activity 

Purpose 

Group 

Size* 

Dura 

■tion 

Course  Snapshots 

Student  Survey,  Start-of-Course 

Measure  NCO  status  at  start  of  ALC 

100-200 

1  hr 

Sit  Judgment  Test,  Start-of-Course 

Measure  NCO  proficiency  at  start  of  ALC 

100-200 

1  hr 

Student  Survey,  End-of-Course 

Measure  NCO  status  at  end  of  ALC 

100-200 

14  hr 

Sit  Judgment  Test,  End-of-Course 

Measure  NCO  proficiency  at  end  of  ALC 

100-200 

1  hr 

Student  Focus  Gp,  End-of-Course 

Capture  reactions  and  expectations,  clarify  trends 

5-6 

2  hr 

End  State  Assessments 

Participant  Survey 

Measure  ALC  impact  at  5-6  months,  obtain  opinions 

10-40 

1  hr 

Participant  Focus  Group 

Expand  understanding,  capture  opinions  and  ideas 

4-6 

2  hr 

Unit  Leader  Survey 

Measure  ALC  impact,  add  perspective 

6-12 

!4  hr 

Unit  Leader  Focus  Group 

Expand  understanding,  capture  opinions  and  ideas 

3-5 

114  hr 

Unit  Peer  Survey 

Measure  ALC  impact,  add  perspective 

6-12 

y2  hr 

Unit  Peer  Focus  Group 

Expand  understanding,  capture  opinions  and  ideas 

4-6 

114  hr 

Records-Based  Capture  Form 

Capture  information  from  unit  records 

N/A 

1-3  hr 

*  Group  sizes  are  approximate  and  may  be  varied  by  applying  professional  judgment. 
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Checklist  1 


SCHEDULING  COURSE  SNAPSHOT  SESSIONS 


Step 

When 

Start  of  Course 

□ 

1 .  Obtain  course  schedule  and  number  of  students 

30  days  prior  to  start 

□ 

2.  Schedule  start-of-course  survey  (1  hr),  large  group*  on  Day  1-2 

30  days  prior  to  start 

□ 

3.  Schedule  start-of-course  SJT  (1  hr),  large  group*  on  Day  1-2 

30  days  prior  to  start 

□ 

4.  Obtain  scheduling  input  and  concurrence  from  Data  Manager 

28  days  prior  to  start 

□ 

5.  Coordinate  schedule  with  course  POC 

25  days  prior  to  start 

□ 

6.  Submit  final  schedule  to  Data  Manager 

1 8  days  prior  to  start 

□ 

7.  Coordinate  data  collection  duties  with  other  data  collector(s) 

14  days  prior  to  start 

□ 

8.  Verify  data  collection  events  on  course  master  schedule 

1 0  days  prior  to  start 

□ 

9.  Verify  classroom  locations  for  data  collection  events 

5  days  prior  to  start 

End  of  Course 

□ 

10.  Schedule  end-of-course  survey  (1/2  hr),  large  group*  on  Day  26 

5  days  after  course  start 

□ 

1 1 .  Schedule  end-of-course  SJT  (1  hr),  large  group*  on  Day  26 

5  days  after  course  start 

□ 

12.  Obtain  current  Master  Roster  from  Data  Manager 

5  days  after  course  start 

□ 

13.  Select  25-30  rostered  participants  in  5  subgroups  of  5-6  each 

5  days  after  course  start 

□ 

14.  Schedule  focus  gps,  1  session  per  by-name  subgroup  (2  hr  ea) 

Target  Days  26-27 

□ 

15.  Obtain  scheduling  input  and  concurrence  from  Data  Manager 

5  days  after  course  start 

□ 

1 6.  Submit  schedule  (including  by-name  list  for  focus  groups)  to 
course  POC  for  approval 

6  days  after  course  start 

□ 

17.  Finalize  schedule  and  deliver  to  Data  Manager 

10  days  after  course  start 

□ 

18.  Coordinate  data  collection  duties  with  other  data  collector(s) 

12  days  after  course  start 

□ 

19.  Verify  data  collection  events  on  course  master  schedule 

15  days  after  course  start 

□ 

20.  Verify  classroom  location  for  each  data  collection  event 

21  days  after  course  start 

*  Schedule  all  students  as  one  large  group  for  surveys  and  SJTs. 
=>  See  notional  schedule  on  next  page  for  example. 
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NOTIONAL  COURSE  SNAPSHOT  SCHEDULE 


Note-1\  Event  durations  and  group  sizes  come  from  Attachment  1 . 

Note-2\  Separate  time  slots  for  back-to-back  survey  and  SJT  sessions  (rather  than  one  longer 
time  slot)  are  recommended  to  facilitate  implementing  time  limits  for  SJT  sessions. 

Note-3:  The  sample  of  students  for  focus  groups  should  be  selected  from  the  Master  Roster  to 
equally  represent  the  different  installations  (for  schoolhouse  cohorts)  or  parent  units  (for  MTT 
cohorts). 

Note-4\  The  survey  session(s)  should  precede  the  first  focus  group  session. 

Note-5:  Two  data  collectors  are  required  for  every  survey,  SJT,  and  focus  group  session. 


Date 

Time 

Event 

Comments 

Start-of-Course  Events 

12  Jan  10 

0900-1000 

Student  Survey  (start  of  course) 

Entire  class  at  once 

1000-1100 

Student  SJT  (start  of  course) 

Entire  class  at  once 

End-of-Course  Events 

4  Feb  10 

0800-0830 

Student  Survey  (end  of  course) 

Entire  class  at  once 

0830-0930 

Student  SJT  (end  of  course) 

Entire  class  at  once 

0930-1130 

Student  Focus  Group  (Group  #1) 

Subgroup  ( n  =  5-6) 

1300-1500 

Student  Focus  Group  (Group  #2) 

Subgroup  ( n  =  5-6) 

1500-1700 

Student  Focus  Group  (Group  #3) 

Subgroup  (n  =  5-6) 

5  Feb  10 

0800-1000 

Student  Focus  Group  (Group  #4) 

Subgroup  ( n  =  5-6) 

1000-1200 

Student  Focus  Group  (Group  #5) 

Subgroup  (n  =  5-6) 
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Checklist  2 


SCHEDULING  END  STATE  SESSIONS  -  MTT  COHORT 


Note-1\  The  Data  Manager  will  schedule  end  state  sessions  for  schoolhouse  cohorts. 
Note-2\  ARI  assumes  that  units  will  require  at  least  90  days  advance  notice  for  scheduling. 
Note-3\  Event  durations  and  group  sizes  come  from  Attachment  1. 

Note-4\  Two  data  collectors  are  required  for  every  focus  group  session. 


Step 

When* 

□ 

1.  Obtain  current  Master  Roster  from  Data  Manager 

1 00  days  prior  to  D-day 

□ 

2.  Compile  list  of  Soldiers  in  cohort,  by  type  (80  participants,  80 
leaders,  80  peers  -  approx) 

1 00  days  prior  to  D-day 

□ 

3.  Schedule  participant  surveys  (1  hr),  in  two  by-name  groups  of  ~40 

1 00  days  prior  to  D-day 

□ 

4.  Schedule  60**  participants  in  2-hr  focus  groups  (4-6  each,  by  name) 

100  days  prior  to  D-day 

□ 

5.  Schedule  leader  surveys  +  focus  groups  (2  hr  combined),  in  12  by¬ 
name  groups  of  6-7  (schedule  all  leaders)*** 

1 00  days  prior  to  D-day 

□ 

6.  Schedule  peer  surveys  +  focus  groups  (2  hr  combined),  in  12  by¬ 
name  groups  of  6-7  (schedule  all  peers)*** 

1 00  days  prior  to  D-day 

□ 

7.  Crosswalk  group  lists  against  cohort  list  to  verify  all  participants, 
leaders,  and  peers  are  scheduled  for  surveys 

1 00  days  prior  to  D-day 

□ 

8.  Schedule  records-based  data  gathering  sessions,  by  HQ  element 
(all  battalion  and  company  HQs  of  cohort  participants) 

1 00  days  prior  to  D-day 

□ 

9.  Obtain  scheduling  input  and  concurrence  from  Data  Manager 

98  days  prior  to  D-day 

□ 

10.  Submit  by-name  schedule  to  unit  POC  for  coordination  &  approval 

95  days  prior  to  D-day 

□ 

1 1 .  Finalize  schedule  and  deliver  to  Data  Manager 

70  days  prior  to  D-day 

□ 

12.  Verify  data  collection  events  on  unit  training  schedule 

50  days  prior  to  D-day 

□ 

13.  Assign  by-session  duties  to  data  collection  teams/members 

30  days  prior  to  D-day 

□ 

14.  Verify  facilities,  equipment,  and  supplies  for  each  event 

15  days  prior  to  D-day 

*  D-day  =  data  start  (first  day  of  actual  data  collection)  for  a  given  cohort. 

**  Select  60  of  80  participants  randomly,  trying  to  represent  the  different  parent  units  evenly. 

***  All  leaders  and  peers  will  complete  surveys,  but  some  in  each  group  will  be  released  (as 
necessary)  at  the  start  of  a  focus  group  to  meet  desired  group  size  (see  Attachment  1 ). 

=>  See  notional  schedule  on  next  page  for  example. 
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NOTIONAL  END  STATE  SCHEDULE  FOR  MTT  COHORT 


Note-1\  See  Attachment  1  for  desired  group  sizes. 

Note-2\  A  survey  should  precede  a  focus  group  for  a  given  group. 

Note-3\  All  survey  sessions  and  focus  groups  require  both  members  of  a  data  collection  team, 
except  when  a  survey  session  involves  fewer  than  25  Soldiers. 


Date 

Time 

Event 

Comments 

26  Jul  10 

0900-1000 

Participant  Survey  (14  of  sample) 

DC  Team  #1 

1000-1200 

Participant  Focus  Group  (Groups  #P1  &  #P2) 

DC  Teams  #1  and  #2 

1300-1400 

Participant  Survey  (14  of  sample) 

DC  Team  #1 

1300-1500 

Participant  Focus  Group  (Group  #P3) 

DC  Team  #2 

1500-1700 

Participant  Focus  Group  (Groups  #P4  &  #P5) 

DC  Teams  #1  and  #2 

27  Jul  10 

0800-1000 

Participant  Focus  Group  (Groups  #P6  &  #P7) 

DC  Teams  #1  and  #2 

1000-1200 

Participant  Focus  Group  (Group  #P8) 

DC  Team  #1 

Leader  Survey  +  Focus  Group  (Group  #L1) 

DC  Team  #2 

1300-1500 

Participant  Focus  Group  (Groups  #P9  &  #P10) 

DC  Teams  #1  and  #2 

1500-1700 

Participant  Focus  Group  (Groups  #P1 1  &  #P12) 

DC  Teams  #1  and  #2 

28  Jul  10 

0800-1000 

Leader  Survey  +  Focus  Group  (Group  s#L2  &  #L3) 

DC  Teams  #1  and  #2 

1000-1200 

Leader  Survey  +  Focus  Group  (Groups  #L4  &  #L5) 

DC  Teams  #1  and  #2 

1300-1500 

Leader  Survey  +  Focus  Group  (Groups  #L6  &  #L7) 

DC  Teams  #1  and  #2 

1500-1700 

Leader  Survey  +  Focus  Group  (Groups  #L8  &  #L9) 

DC  Teams  #1  and  #2 

29  Jul  10 

0800-1000 

Leader  Survey  +  Focus  Group  (Groups  #L10  &  #L1 1) 

DC  Teams  #1  and  #2 

1000-1200 

Leader  Survey  +  Focus  Group  (Group  #L12) 

DC  Team  #1 

Peer  Survey  +  Focus  Group  (Group  #Pr1) 

DC  Team  #2 

1300-1500 

Peer  Survey  +  Focus  Group  (Groups  #Pr2  &  #Pr3) 

DC  Teams  #1  and  #2 

1500-1700 

Peer  Survey  +  Focus  Group  (Groups  #Pr4  &  #Pr5) 

DC  Teams  #1  and  #2 

30  Jul  10 

0800-1000 

Peer  Survey  +  Focus  Group  (Groups  #Pr6  &  #Pr7) 

DC  Teams  #1  and  #2 

1000-1200 

Peer  Survey  +  Focus  Group  (Groups  #Pr8  &  #Pr9) 

DC  Teams  #1  and  #2 

1300-1500 

Peer  Survey  +  Focus  Group  (Groups  #Pr10  &  #Pr1 1) 

DC  Teams  #1  and  #2 

1500-1700 

Peer  Survey  +  Focus  Group  (Group  #Pr12) 

DC  Team  #1 

Unit  Records  Data  Collection 

DC  Team  #2 
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Checklist  3 


ADMINISTERING  A  SURVEY  SESSION 


Note-1\  Two  data  collectors  should  handle  large-group  sessions  (more  than  50  Soldiers). 
Note-2\  All  survey  sessions  will  involve  by-name  groups,  except  the  course  snapshot  surveys. 


Step 

When 

Before  the  Session 

□ 

1.  Assemble  information  about  group  from  approved  schedule 

30  days  prior  to  session 

□ 

2.  Determine  type  and  number  of  Soldiers  in  group 

30  days  prior  to  session 

□ 

3.  Obtain  hardcopies  (including  3  extras)  of  specific  questionnaire 
from  Data  Manager’s  office 

30  days  prior  to  session 

□ 

4.  Double  check  every  hardcopy  for  errors  (missing  pages,  etc.) 

21  days  prior  to  session 

□ 

5.  Ready  by-name  list  of  Soldiers  scheduled  in  survey  group 

7  days  prior  to  session 

□ 

6.  Confirm  location  and  Soldiers  (by  name)  for  survey  session 

7  days  prior  to  session 

□ 

7.  Obtain  pencils  or  pens  to  take  to  survey  session 

5  days  prior  to  session 

□ 

8.  On  every  hardcopy,  fill  in  fields  reserved  for  research  team’s  use 

1  day  prior  to  session 

□ 

9.  Review  questionnaire  so  you  can  answer  Soldiers’  questions 

1  day  prior  to  session 

During  the  Session 

□ 

10.  Verify  that  Soldiers  present  match  list  by  name 

Start  of  session 

□ 

1 1 .  Introduce  the  survey  and  its  role  in  ARI’s  evaluation 

Start  of  session 

□ 

12.  Hand  out  hardcopy  questionnaires,  1  per  Soldier 

Start  of  session 

□ 

13.  Review  the  questionnaire’s  instructions 

Start  of  session 

□ 

14.  Explain  privacy  protections,  especially  the  ID# 

Start  of  session 

□ 

15.  Instruct  group  to  stay  for  follow-on  focus  group  (if  appropriate) 

Start  of  session 

□ 

16.  Remain  with  Soldiers  as  they  work  individually 

As  Soldiers  work 

□ 

17.  Answer  questions  raised  by  Soldiers 

Throughout 

□ 

18.  Collect  completed  questionnaires  as  Soldiers  finish 

End  of  session 

□ 

19.  Verify  completeness  of  admin  info  on  every  questionnaire 

End  of  session 

□ 

20.  Remind  Soldiers  to  stay  for  follow-on  focus  group  (if  appropriate) 

End  of  session 

□ 

21 .  Release  individuals  after  verifying  their  admin  info 

End  of  session 
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Checklist  4 


ADMINISTERING  A  SITUATIONAL  JUDGMENT  TEST 


Note-1\  Two  data  collectors  should  work  together  due  to  large  group  sizes. 

Note-2\  All  students  will  complete  both  SJT  sessions  (start-of-course  and  end-of-course). 


Step 

When 

Before  the  Session 

□ 

1.  Assemble  information  about  group  from  class  roster 

30  days  prior  to  session 

□ 

2.  Determine  number  of  students  in  group 

30  days  prior  to  session 

□ 

3.  Obtain  hardcopies  (including  5  extras)  of  correct  SJT  form  (start- 
of-course  or  end-of-course)  from  Data  Manager’s  office 

30  days  prior  to  session 

□ 

4.  Double  check  every  hardcopy  for  errors  (missing  pages,  etc.) 

21  days  prior  to  session 

□ 

5.  Confirm  location  (and  equipment,  if  needed)  for  session 

7  days  prior  to  session 

□ 

6.  Obtain  pencils  or  pens  to  take  to  SJT  session 

5  days  prior  to  session 

□ 

7.  On  every  hardcopy,  fill  in  fields  reserved  for  research  team’s  use 

1  day  prior  to  session 

□ 

8.  Review  SJT  form  so  you  can  answer  students’  questions 

1  day  prior  to  session 

During  the  Session 

□ 

9.  Perform  a  head  count  to  determine  completeness  of  group 

Start  of  session 

□ 

10.  Introduce  the  SJT  and  its  role  in  ARI’s  evaluation 

Start  of  session 

□ 

1 1 .  Hand  out  hardcopy  SJT  forms,  1  per  student 

Start  of  session 

□ 

12.  Review  SJT  form’s  instructions,  including  the  time  limit 

Start  of  session 

□ 

13.  Explain  privacy  protections,  especially  the  ID# 

Start  of  session 

□ 

14.  Instruct  subgroup  to  stay  for  follow-on  focus  group  (if  appropriate) 

Start  of  session 

□ 

15.  Track  time,  monitor  students  to  ensure  they  work  individually 

As  students  work 

□ 

16.  Answer  questions  raised  by  students 

Throughout 

□ 

17.  Collect  completed  SJT  forms  as  students  finish 

End  of  session 

□ 

18.  Verify  completeness  of  admin  information  on  every  form 

End  of  session 

□ 

19.  Announce  end  of  session  when  time  limit  is  reached 

End  of  session 

□ 

20.  Remind  subgroup  to  stay  for  follow-on  focus  group  (if  appropriate) 

End  of  session 

□ 

21.  Release  individuals  after  verifying  their  admin  information 

End  of  session 
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Checklist  5 


CONDUCTING  A  FOCUS  GROUP  SESSION 


Note-1\  Two  data  collectors  (facilitator  +  note  taker)  will  conduct  a  focus  group. 

Note-2\  The  note  taker  will  enter  notes  directly  into  a  laptop  and  digitally  record  the  dialogue. 
Note-3\  The  typical  focus  group  will  immediately  follow  a  survey  session. 

Note-4\  A  data  collection  team  should  conduct  no  more  than  4  focus  groups  in  a  day. 


Step 

When 

Before  the  Session 

□ 

1.  Identify  type  of  focus  group  (e.g.,  peers)  from  approved  schedule 

9  days  prior  to  session 

□ 

2.  Print  a  hardcopy  of  the  specific  protocol  (e.g.,  leader  focus  group) 

9  days  prior  to  session 

□ 

3.  Follow  advance  preparation  instructions  on  protocol 

As  directed  in  protocol 

□ 

4.  Ready  by-name  list  of  Soldiers  scheduled  for  the  session 

7  days  prior  to  session 

□ 

5.  Confirm  location  and  Soldiers  (by  name)  for  the  session 

7  days  prior  to  session 

□ 

6.  Coordinate  specific  duties  with  other  data  collector 

5  days  prior  to  session 

□ 

7.  Ready  voice  recorder  (including  extra  batteries)  and  laptop 

1  day  prior  to  session 

□ 

8.  Annotate  the  protocol,  prioritize  questions,  do  a  mental  walk-thru 

1  day  prior  to  session 

During  the  Session 

□ 

9.  Verify  that  Soldiers  on  hand  match  list  by  name 

Start  of  session 

□ 

10.  Release  extra  Soldiers,  if  necessary,  to  reach  desired  group  size 

Start  of  session 

□ 

11.  Introduce  the  session  and  its  role  in  ARI’s  evaluation 

Start  of  session 

□ 

12.  Explain  privacy  protections  and  voice  recording 

Start  of  session 

□ 

13.  Start  voice  recorder  &  state  installation/group/facilitator/date/time 

Start  of  dialogue 

□ 

14.  Follow  protocol’s  instructions  for  facilitating  and  taking  notes 

Throughout  dialogue 

□ 

15.  Monitor  clock  to  ensure  that  session  ends  on  time 

Throughout  dialogue 

□ 

16.  Turn  off  voice  recorder 

End  of  session 

□ 

17.  Release  Soldiers  after  session  wrap-up 

End  of  session 

After  the  Session 

□ 

18.  Transfer  voice  file  to  laptop  or  flash  drive,  with  standard  file  name 

After  release  of  Soldiers 

□ 

19.  Document  procedural  exceptions  for  incorporation  into  notes 

After  release  of  Soldiers 

□ 

20.  Flesh  out  digital  notes  in  Microsoft  Word  file  (note  taker) 

NLT  end  of  same  day 

□ 

21 .  Review  notes  (facilitator)  and  finalize  them  with  aid  of  voice  file 

Within  3  days 
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Checklist  6 


COLLECTING  DATA  FROM  UNIT  RECORDS 


Note-1\  One  data  collector  can  work  alone  when  gathering  data  from  unit  records. 

Note-2\  Data  collectors  will  treat  all  unit  records-based  data  as  sensitive  and  FOUO. 

Note-3\  Some  units  may  choose  to  provide  data  elements  (extracts)  rather  than  records. 
Note-4\  Duration  of  a  session  may  vary  considerably,  depending  on  specific  circumstances. 


Step 

When 

Before  the  Session 

□ 

1 .  Identify  records  targeted  (Bn  or  Co)  from  approved  schedule 

15  days  prior  to  session 

□ 

2.  Determine  HQ,  POC,  and  location  for  the  session 

15  days  prior  to  session 

□ 

3.  Coordinate  with  POC  regarding  data  needed  and  ground  rules 

15  days  prior  to  session 

□ 

4.  Ready  list  of  participants  and  squads  falling  under  the  HQ 

7  days  prior  to  session 

□ 

5.  Send  to  POC  the  list  of  participants/squads  +  capture  form 

7  days  prior  to  session 

□ 

6.  Confirm  appointment  with  POC 

2  days  prior  to  session 

□ 

7.  Print  the  Records-Based  Capture  Form 

1  day  prior  to  session 

□ 

8.  Become  thoroughly  familiar  with  the  capture  form’s  data  elements 

1  day  prior  to  session 

During  the  Session 

□ 

9.  Introduce  ARI’s  evaluation  and  need  for  records-based  data 

Start  of  session 

□ 

10.  Explain  privacy  protections  and  handling  of  data 

Start  of  session 

□ 

1 1 .  Reach  agreement  with  POC  on  specific  procedures 

Start  of  session 

□ 

12.  Verify  names  and  positions  of  participants  on  the  list 

Start  of  session 

□ 

13.  Record  data  elements  on  capture  form;  note  missing  data  &  why 

As  session  progresses 

□ 

14.  Review  each  page  of  capture  form  for  completeness 

End  of  session 

□ 

15.  Provide  POC  a  list  of  data  elements  expected  later,  if  any 

End  of  session 

□ 

16.  Answer  POC’s  final  questions  and  express  ARI’s  appreciation 

End  of  session 

After  the  Session 

□ 

17.  Add  notes  to  capture  form  re:  irregularities,  causes  of  missing 
data,  and  pending  data 

NLT  end  of  same  day 

□ 

18.  Place  capture  form  in  envelope  or  enclosed  folder  marked  FOUO 

NLT  end  of  same  day 

□ 

19.  Document  procedural  exceptions  for  submission  to  Data  Manager 

Within  2  days 

□ 

20.  Obtain  pending  data  from  POC  and  add  to  capture  form 

Within  5  days 

□ 

21 .  Label  capture  form  as  “final/complete”  or  “closed/incomplete” 

Within  7  days 
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Checklist  7 


SUBMITTING  MATERIALS  TO  DATA  MANAGER 


Note-1\  As  a  general  rule,  data  from  one  site  visit  will  be  packaged  together  and  hand-carried  to 
the  Data  Manager. 

Note-2\  If  hardcopy  originals  must  be  shipped  to  the  Data  Manager,  backups  will  first  be 
scanned  into  digital  files  or  photocopied. 


Step 

When* 

□ 

1.  Assemble  all  data  collected  during  the  site  visit,  which  may  include: 

•  Questionnaires  completed  by  Soldiers 

•  SJT  forms  completed  by  students 

•  Final  notes  from  focus  groups 

•  Digital  voice  files  from  focus  groups  (if  requested) 

•  Records-based  capture  forms  completed  by  data  collectors 

•  Documentation  of  departures  from  standard  procedures 

•  Lessons  learned  by  data  collection  team(s) 

Within  3  days  after  C-day 

□ 

2.  Verify  administrative  information  and  completeness  of  data  package 

Within  5  days  after  C-day 

□ 

3.  Complete  annotations  on  Records-Based  Capture  Forms 

Within  5  days  after  C-day 

□ 

4.  Confirm  that  names  of  digital  files  meet  formatting  standard 

Within  7  days  after  C-day 

□ 

5.  Compile  documentation  of  departures  from  standard  procedures 

Within  7  days  after  C-day 

□ 

6.  Compile  data  collectors'  lessons  learned  into  Microsoft  Word  file 

Within  9  days  after  C-day 

□ 

7.  Email  final  digital  files  (minus  voice  files)  to  Data  Manager 

Within  9  days  after  C-day 

□ 

8.  Prepare  inventory  of  hardcopy  and  digital  materials  in  package 

Within  9  days  after  C-day 

□ 

9.  Copy  final  digital  files  +  inventory  onto  CD,  and  label  CD  (e.g.,  Ft 
Hood  Data,  5-1 3  Jan  201 0) 

Within  10  days  after  C-day 

□ 

10.  Print  hardcopy  of  data  package’s  inventory 

Within  10  days  after  C-day 

□ 

1 1 .  Package  inventory  +  hardcopy  materials  +  CD  and  hand  carry  to 
Data  Manager 

Within  10  days  after  C-day 

If  the  Data  Manager  directs  you  to  ship  a  data  package: 

□ 

12.  Before  packaging,  scan  1  backup  copy  of  each  completed 
questionnaire  +  SJT  form  +  records-based  capture  form 

Prior  to  shipping 

□ 

13.  Coordinate  shipping  arrangements  with  Data  Manager 

2  days  before  shipping 

□ 

14.  Notify  Data  Manager  when  data  package  has  been  shipped 

W/in  4  hours  after  shipping 

□ 

15.  Confirm  that  Data  Manager  received  data  package 

W/in  2  days  after  shipping 

*  C-day  =  completion  of  site  visit  (last  day). 


1-14 


Checklist  8 


PROTECTING  AND  STORING  DATA  LOCALLY 


Note-1\  While  holding  data  temporarily  or  long-term,  each  data  collection  team  must  protect  it 
from  unauthorized  access  and  store  it  properly.  This  includes  hardcopy  and  digital  data. 

Note-2\  Locally  held/stored  data  will  be  organized  by  site  visit,  one  data  package  per  visit. 

Note-3\  No  personnel  outside  the  research  team  will  have  access  to  any  data. 

Note-4\  No  hardcopy  or  digital  data  will  be  destroyed  unless  the  Data  Manager  directs  the  data 
collection  team  to  do  so. 


Step 

When* 

□ 

1 .  Create  the  structure  for  storing  and  tracking  data: 

•  File  cabinet  or  boxes  for  hardcopy  materials  (in  locked  area) 

•  Digital  folders  for  organizing  related  files  (password  protected) 

•  External  device  for  storing  digital  files  (access  protected) 

•  Digital  inventory  for  tracking  all  data  stored  locally 

Start-up  phase 

□ 

2.  Place  records-based  capture  forms  in  envelope  or  enclosed  folder 
prominently  marked  “FOUO” 

Day  of  collection 

□ 

3.  Place  all  hardcopy  materials  from  a  site  visit  together,  in  folders 
identifying  type  and  source  (e.g.,  peer  survey  Ft  Hood  #1 ) 

Within  7  days  after  C-day 

□ 

4.  Make  sure  every  digital  file  name  is  complete  and  unique 

Within  7  days  after  C-day 

□ 

5.  Place  all  digital  files  in  site  visit-specific  folder,  using  subfolders: 

•  Final  notes  from  focus  groups  (discard  interim  files) 

•  Digital  voice  files  from  focus  groups 

•  Data  collector  notes  (procedural  exceptions,  lessons  learned,  etc.) 

Within  7  days  after  C-day 

□ 

6.  Discard  handwritten  notes,  if  any,  from  focus  groups 

After  digital  files  finalized 

□ 

7.  Externally  backup  site  visit  data  package  when  it  is  completed 

Within  8  days  after  C-day 

□ 

8.  Maintain  administrative  records,  including  shipment  &  delivery  dates 

Within  14  days  after  C-day 

□ 

9.  Continuously  control  access  to  hardcopy  data  and  external  media 
(locked  container  or  area)  and  computer  files  (password  protected) 

24x7 

□ 

10.  Refer  all  Soldier  or  unit  requests  for  data  to  Evaluation  Manager 

As  requests  arise 

□ 

11.  When  directed  by  Data  Manager,  destroy  (shred)  hardcopy  data 

When  directed 

□ 

12.  Annotate  inventory  re:  destroyed  data  (date  and  authority) 

Upon  destruction  of  data 

□ 

13.  At  end  of  project,  deliver  all  retained  materials  to  Data  Manager 

Close-out  phase 

*  C-day  =  completion  of  site  visit  (last  day). 
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